Hello everyone,
I conducted a factorial survey (vignette study = hypothetical scenario description evaluated by participants). In this study, each participant evaluated three different vignettes (scenario accepted or not). A scenario always consists of two dimensions (also attributes) with three characteristics (also levels) each.
(1) Now I would like to calculate the effects of the vignette content/levels shown on the vignette evaluation. First, I modeled a (single-level) logistic regression with main effects of the vignette levels and additional interaction effects between the level content and participant characteristics (age, gender, etc.). My final model has a final -2 log likelihood of 868.461 (from an initial 985.541 in the model with only main effects).
(2) Since the data is nested with three ratings per participant, I now want to calculate a two-level logistic regression (random intercept) with vignette characteristics at level 1 and respondents’ characteristics at level 2; with one random intercept and all other predictors in the model included as fixed effects.
My final -2 log-likelihood in the two-level regression is now much higher at 3,157.793. According to the output of the covariance parameter overview, I have estimated 25 fixed effects and one random effect. However, my model only includes 12 dichotomous fixed effects. Why are the two values of the dichotomous variables (with one being 0) counted as two effects? Is the high number of fixed effects/parameters the reason for the high -2 log-likelihood? Is the model penalised for this?
(3) The random intercept in our two-level null model is small (0.099, p=0.238), indicating that we do not have significant variability at the intercept, suggesting that we do not have any clustering going on in the data, right? The interclass correlation coefficient (ICC) in your study in the null model is also small (0.024), indicating that the chance of accepting a vignette does not differ from one participant to another.
Do these three facts ((1) higher -2log-likelihood than in the simple 1-level model, (2) no significant variability on the intercept, and (3) smaller ICC) suggest not performing a two-level analysis and instead reporting the parameter estimates on the simple (1-level) logistic regression?
Thank you very much in advance for your help.
Best, Julia