I have a question about the differences between the output in the tests of model effects and the parameter estimates in generalized linear models SPSS v. 27. We used negative binomial regression because our outcome (# of days of poor mental health) is a count variable. Our models include 4 dichotomous effects-coded demographic variables (e.g., gender) and one variable with 4 categories (race), group (whether data were collected in 2018 vs. 2020), the interactions between each demographic variable and group, and two covariates (campus and year in school). Looking a the output, in the tests of model effects and the parameter estimates the Wald chi square tests and p values differ. We are trying to figure out why that is given that the same terms are in the models. The tests of model effects use Type 3 tests; we can't figure out what tests are used for the parameter estimates. The Wald chi square tests are the same for the interaction terms across the two outputs. Thank you for any help you can provide.

------------------------------
Patricia Frazier
------------------------------

#SPSSStatistics

Hi, Patricia. I sent a message to our statisticians and one of them responded with this:

"The Wald Chi-Square in the Parameter Estimates table is the square of the Wald z-statistic which could be obtained by calculating B/Std.Error (The first two columns after Parameter). Its associated df is always 1. However, in the Tests of Model Effects table, each effect corresponds to one test (one row). The df depends on the levels. The Wald Chi-Square statistics in the Tests of Model Effects are supposed to match for covariates, but not for factors with more than 1 level. "

------------------------------
Rick Marcantonio
Quality Assurance
IBM
------------------------------

Original Message:
Sent: Thu May 20, 2021 02:20 PM
From: Patricia Frazier
Subject: tests of model effects vs. parameter estimates in generalized linear model output

I have a question about the differences between the output in the tests of model effects and the parameter estimates in generalized linear models SPSS v. 27. We used negative binomial regression because our outcome (# of days of poor mental health) is a count variable. Our models include 4 dichotomous effects-coded demographic variables (e.g., gender) and one variable with 4 categories (race), group (whether data were collected in 2018 vs. 2020), the interactions between each demographic variable and group, and two covariates (campus and year in school). Looking a the output, in the tests of model effects and the parameter estimates the Wald chi square tests and p values differ. We are trying to figure out why that is given that the same terms are in the models. The tests of model effects use Type 3 tests; we can't figure out what tests are used for the parameter estimates. The Wald chi square tests are the same for the interaction terms across the two outputs. Thank you for any help you can provide.

------------------------------
Patricia Frazier
------------------------------

#SPSSStatistics

Hi again. A second statistician responded with this:

"As the previous response notes, in some cases the estimates don't correspond to the same overall effects as are in the Model Effects tables. This would be the case with multiple degree of freedom tests. In the case of single degree of freedom model effects tests, the issue is probably the same thing discussed in this copied Technote (although it discusses a linear model, the same issues arise for generalized linear models, and applies pretty much any time you have interactions involving factors):

Problem

I'm running a linear model in GLM or UNIANOVA, with a 2x2 factorial design. I'm comparing the t tests for the parameter estimates to the F tests in the ANOVA table for the same effects and finding that the significance values differ. Why is this?

Resolving The Problem

The reason that the F tests in the ANOVA table in GLM and UNIANOVA don't give the squares of the t values for the coefficients and the same significances (as they would if they were testing identical hypotheses) is that the Type III hypotheses on which the Type III sums of squares are based are not the same as the hypotheses tested for the coefficients. You can use the SPSS MATRIX procedure to see more clearly what the coefficients that are estimated represent, by running the following commands:

matrix.
compute X={1,1,1,1;
1,0,1,0;
1,1,0,0;
1,0,0,0}.
compute C=inv(sscp(X))*t(X).
print X.
end matrix.

X here is the design matrix at the cell level for a 2x2 design. C is the contrast matrix produced by the design matrix X, which is:

0 0 0 1
0 0 1 -1
0 1 0 -1
1 -1 -1 1

This shows that the first parameter estimate gives the mean value in the fourth cell of the design. The second estimate is of the mean for the third cell minus that for the fourth cell. The third estimate is that for the second cell minus the fourth. The fourth estimate is the sum of the first and fourth cell means minus the sum of the second and third.

It should be clear that these estimates as given do not (aside from the last one) represent particularly informative quantities for purposes of evaluating the 2x2 design in terms of main effects and interactions. Only the interaction estimate is actually giving what one might expect. The reason for this is that the interaction in this model is not contained in any higher order effects and is thus estimable, while the main effects in the model are contained in the interaction and are not estimable. This means that there is no pure main effect in this model for us to estimate, and we have to choose what function of the model parameters we want to use to test what we call a "main effect" (if we even decide to do so; technically it makes more sense to avoid discussing main effects if a containing interaction is present and to discuss simple main effects or other functions that take the interaction into account).

The most common choice of a linear function of the parameters to use to represent a "main effect" in such a design produces what might be called an averaged main effect. This compares marginal means for the levels of the factor of interest, averaging over the levels of the other factor in an unweighted manner (weighted vs. unweighted being relevant only with unequal N). In a design like this with no empty cells, the Type III hypotheses tested by the Type III sums of squares in the ANOVA table give these particular versions of "main effects" for the two factors."

------------------------------
Rick Marcantonio
Quality Assurance
IBM
------------------------------

Original Message:
Sent: Thu May 20, 2021 02:20 PM
From: Patricia Frazier
Subject: tests of model effects vs. parameter estimates in generalized linear model output

I have a question about the differences between the output in the tests of model effects and the parameter estimates in generalized linear models SPSS v. 27. We used negative binomial regression because our outcome (# of days of poor mental health) is a count variable. Our models include 4 dichotomous effects-coded demographic variables (e.g., gender) and one variable with 4 categories (race), group (whether data were collected in 2018 vs. 2020), the interactions between each demographic variable and group, and two covariates (campus and year in school). Looking a the output, in the tests of model effects and the parameter estimates the Wald chi square tests and p values differ. We are trying to figure out why that is given that the same terms are in the models. The tests of model effects use Type 3 tests; we can't figure out what tests are used for the parameter estimates. The Wald chi square tests are the same for the interaction terms across the two outputs. Thank you for any help you can provide.

------------------------------
Patricia Frazier
------------------------------

#SPSSStatistics

Thank you for the replies. As I understand it, the parameter estimates are not very interpretable for the main effects and we may be better off using the tests of model effects. Please correct me if I am wrong. If I am correct, I have a couple follow-up questions.
1) We did 5 imputations but the pooled results do not appear for the tests of model effects. Is there a way to get those?
2) Do the significance tests in the tests of model effects apply to the estimated marginal means later in the output (which do not have associated p values)? See output below. Thanks in advance.




------------------------------
Patricia Frazier
------------------------------

Original Message:
Sent: Thu May 20, 2021 06:40 PM
From: Rick Marcantonio
Subject: tests of model effects vs. parameter estimates in generalized linear model output

Hi again. A second statistician responded with this:

"As the previous response notes, in some cases the estimates don't correspond to the same overall effects as are in the Model Effects tables. This would be the case with multiple degree of freedom tests. In the case of single degree of freedom model effects tests, the issue is probably the same thing discussed in this copied Technote (although it discusses a linear model, the same issues arise for generalized linear models, and applies pretty much any time you have interactions involving factors):

Problem

I'm running a linear model in GLM or UNIANOVA, with a 2x2 factorial design. I'm comparing the t tests for the parameter estimates to the F tests in the ANOVA table for the same effects and finding that the significance values differ. Why is this?

Resolving The Problem

The reason that the F tests in the ANOVA table in GLM and UNIANOVA don't give the squares of the t values for the coefficients and the same significances (as they would if they were testing identical hypotheses) is that the Type III hypotheses on which the Type III sums of squares are based are not the same as the hypotheses tested for the coefficients. You can use the SPSS MATRIX procedure to see more clearly what the coefficients that are estimated represent, by running the following commands:

matrix.
compute X={1,1,1,1;
1,0,1,0;
1,1,0,0;
1,0,0,0}.
compute C=inv(sscp(X))*t(X).
print X.
end matrix.

X here is the design matrix at the cell level for a 2x2 design. C is the contrast matrix produced by the design matrix X, which is:

0 0 0 1
0 0 1 -1
0 1 0 -1
1 -1 -1 1

This shows that the first parameter estimate gives the mean value in the fourth cell of the design. The second estimate is of the mean for the third cell minus that for the fourth cell. The third estimate is that for the second cell minus the fourth. The fourth estimate is the sum of the first and fourth cell means minus the sum of the second and third.

It should be clear that these estimates as given do not (aside from the last one) represent particularly informative quantities for purposes of evaluating the 2x2 design in terms of main effects and interactions. Only the interaction estimate is actually giving what one might expect. The reason for this is that the interaction in this model is not contained in any higher order effects and is thus estimable, while the main effects in the model are contained in the interaction and are not estimable. This means that there is no pure main effect in this model for us to estimate, and we have to choose what function of the model parameters we want to use to test what we call a "main effect" (if we even decide to do so; technically it makes more sense to avoid discussing main effects if a containing interaction is present and to discuss simple main effects or other functions that take the interaction into account).

The most common choice of a linear function of the parameters to use to represent a "main effect" in such a design produces what might be called an averaged main effect. This compares marginal means for the levels of the factor of interest, averaging over the levels of the other factor in an unweighted manner (weighted vs. unweighted being relevant only with unequal N). In a design like this with no empty cells, the Type III hypotheses tested by the Type III sums of squares in the ANOVA table give these particular versions of "main effects" for the two factors."

------------------------------
Rick Marcantonio
Quality Assurance
IBM

Original Message:
Sent: Thu May 20, 2021 02:20 PM
From: Patricia Frazier
Subject: tests of model effects vs. parameter estimates in generalized linear model output

I have a question about the differences between the output in the tests of model effects and the parameter estimates in generalized linear models SPSS v. 27. We used negative binomial regression because our outcome (# of days of poor mental health) is a count variable. Our models include 4 dichotomous effects-coded demographic variables (e.g., gender) and one variable with 4 categories (race), group (whether data were collected in 2018 vs. 2020), the interactions between each demographic variable and group, and two covariates (campus and year in school). Looking a the output, in the tests of model effects and the parameter estimates the Wald chi square tests and p values differ. We are trying to figure out why that is given that the same terms are in the models. The tests of model effects use Type 3 tests; we can't figure out what tests are used for the parameter estimates. The Wald chi square tests are the same for the interaction terms across the two outputs. Thank you for any help you can provide.

------------------------------
Patricia Frazier
------------------------------
#SPSSStatistics