I have a question regarding the interpretation of the results from an experiment I conducted. Each participant answered 4 questions measuring motivation, satisfaction, help, and collaboration (my dependent variables) in 7 different scenarios (my independent variables). To analyze my results, I used three methods: a Wilcoxon test, a regression with standard errors clustered at the individual level - CRSE (to control for individual heterogeneity), and an ordinal regression (using GENLIN ) to account for the ordinal nature of the dependent variable.
Why this approach :
To address the limitations of IBM SPSS, which has a restricted range of robust methods (Field, 2018), Huang's (2020) SPSS macro was utilized. This macro was specifically designed to provide cluster-robust standard error (CRSE) estimates, which is crucial for analyses in which observations are not independent of each other. Using this macro, the aim was to adjust the standard errors to account for the clustering structure of the data at the individual level, ensuring more reliable results in the analysis.
I also conducted ordinal regressions, using GENLIN procedure on SPSS, to account for the dependent variables' ordinality. This type of regression seems well suited for my study, as dependent variables are also ordinal. The participants' responses in this study were based on a 7-point Likert scale, ranging from "strongly disagree" to "strongly agree".
My goal was to : compare the results obtained from the Wilcoxon signed-rank tests with the regressions with clustered standard errors at the individual level to the results and as a confirmatory analysis use the the ordinal regressions to ensure that the conclusions about the impact of different scenarios on the dependent variables are consistent. The goal was to check for the robustness of the Wilcoxon test results. If the conclusions from the different tests diverge, it suggests that the observed effect is less reliable. Conversely, if the effect is significant in all three tests, it is robust
The aim of this analysis was to verify if the significant results obtained with the Wilcoxon test were consistent across the other two methods. I conclude that significant results found with the Wilcoxon test, if they are also significant in the other two regressions, are robust.
Conversely, if an effect is significant in the Wilcoxon test and in the regression with CRSE ( standard errors cluster at the individual level), but not in the ordinal regression (GENLIN ordinal), I consider that this is not a robust effect, indicating that the result is not consistent across the three tests, this indicates that there is an indication of the effect, but that this indication is weak.
I am wondering how to properly interpret this ? What does it really mean ?
For the majority of my results, they are robust, but I have some scenarios where significant effects on certain dependent variables are no longer significant in the ordinal regression, but are in the Wilcoxon test and the regression with clustered standard errors. I am wondering why this happens and how to explain it.
I am working with SPSS version 27. Could you help me better understand these results and their interpretation?
Thank you in advance for your help.
------------------------------
Emma Iannello
------------------------------