# Exercise 2.1, step 5: Interpretation of the output

Now comes the most important step of the analysis: the interpretation of the output. The output can be accessed by clicking the ‘View text’ button.

The estimation has produced a massive amount of output. The AMOS ‘Output window’ allows you to navigate through this output in a very efficient way by selecting different parts of the output, different groups and different models.

We start by inspecting the fit of the different models. Click ‘Model fit’, and then ‘CMIN’. The ‘CMIN’ heading contains the chi-square values and the number of degrees of freedom for the various models. Remember that we are only interested in three models: unconstrained (configural equivalence), measurement weights (metric equivalence) and measurement intercepts (scalar equivalence).

### Question

What do these chi-square values tell you about the fit of the models?

SolutionThe unconstrained model has a chi-square value of 0. This is not surprising - models with only three indicators and without additional constraints are just-identified (this means that they have 0 degrees of freedom). Fit indices for this model are therefore not very informative. The ‘measurement weights’ model has a chi-square value of 1.573,024 for 100 degrees of freedom; the measurement incepts model has a chi-square value of 15.536,361 for 250 degrees of freedom. The higher the chi-square value, the larger the discrepancy between our model and the observed data. The p-values for both models are 0. This indicates that it is very unlikely that the model will give an adequate description of the data. In other words, the chi-square tests indicate bad model fit and reject metric as well as scalar equivalence. However, chi-square tests are known to be very sensitive to large sample sizes. If the sample size is sufficiently large, even negligible misspecifications in the model may lead to the rejection of the whole model. Because we are analysing a massive sample size here (over 80,000 respondents!), it is better not to pay too much attention to these chi-square tests.

To avoid the problems of these chi-square tests, alternative indices of model fit have been developed. Root Mean Squared Error of Approximation (RMSEA) and Comparative Fit Index (CFI) are two very informative measures of how close the model corresponds with the data. Click on ‘Baseline Comparisons’ and ‘RMSEA’ to view this alternative measure of model fit.

### Question

What do you conclude based on the CFI and RMSEA values?

SolutionAs a rule of thumb, RMSEA values lower than 0.05 are considered to indicate acceptable model fit. Judging by RMSEA, the ‘measurement weights’ and the ‘measurements intercepts’ models fit the data reasonably well. Generally, it is assumed that models with CFI values larger than 0.90 are acceptable. Thus, the CFI-criterion also points in the direction of reasonable model fit, although the CFI of the ‘measurement intercepts’ model only marginally exceeds 0.90.

Overall fit indices such as RMSEA and CFI thus seem to provide some evidence that the REJECT-scale is comparable across countries and time points. However, to test measurement equivalence, we should not rely completely on RMSEA and CFI. RMSEA and CFI are measures of overall model fit. They summarise the goodness-of-fit of a complete model in a single number. The model could, for example, contain a severe misspecification in one of the groups or for one specific parameter, but still have a reasonable overall fit. Therefore, we should also check that no misfit is present in various parts of the model separately. In our case, a thorough test of measurement equivalence requires that we look at possible misfit due to the constrained factor loadings and intercepts. So-called ‘modification indices’ are a very useful tool for inspecting for local misfit. For all constrained parameters in the model, AMOS calculates a modification index. Modification indices indicate how much the chi-square value of a model would drop if the parameter were free instead of constrained (in other words, by how much the model fit would improve). Modification indices are in fact chi-square tests for individual equality constraints: high values indicate that the respective parameter constraint is ‘wrong’. However, since chi-square values are known to be very sensitive to large sample sizes (cf. supra), only really high values (larger than 20, say) should be taken as serious evidence of misfit. Because the chi-square values can be misleading, it is advisable to look at the expected parameter change as well. The parameter change indicates how much a parameter would change if the equality constraint were removed. Obviously, we are only interested in parameter changes that are substantial (larger than 0.10 in absolute value, say).

To summarise, a conclusive test of measurement equivalence can be performed by looking at modification indices and expected parameter changes for the factor loadings and measurement intercepts. We start by looking at the equality constraints for the factor loadings. Click ‘modification indices’ and then ‘measurement weights (as mentioned above, these are the factor loadings). Next, select the ‘Measurement intercepts’ model in the bottom left corner of the output window, (this is the model with the constrained factor loadings and intercepts). Now you see modification indices and parameter changes for the first group (AT1). Not all the modification indices presented refer to factor loadings - you should only pay attention to relations between items and the latent variable F1 (these modification indices are indicated by a statement with the following structure: ‘item name <--- F1’). Modification indices for the other groups can be examined by scrolling through the groups in the left-hand column.

- Select 'Modification indices - Regression weights'
- Select 'Measurement intercepts' model
- Scroll through the groups

### Question

Inspect modification indices and parameter changes for the factor loadings in each of the 51 groups. Do you see violated equality constraints (i.e. a modification index larger than 20 and a parameter change larger than 0.10 in absolute value)? If so, in which groups and for which items?

SolutionQuite a few equality constraints on the factor loadings appear to be violated. An overview of violations can be found in the table below:

Group | Parameter | Modification Index | Parameter change |
---|---|---|---|

AT1 | imsmetn | 33.94 | 0.107 |

CH2 | imsmetn | 23.343 | -0.112 |

DK1 | imsmetn | 47.884 | -0.2 |

DK2 | imsmetn | 112.224 | -0.357 |

DK2 | imdfetn | 22.905 | 0.108 |

DK3 | imsmetn | 74.691 | -0.309 |

DK3 | imdfetn | 32.185 | 0.131 |

ES1 | imsmetn | 45.221 | 0.109 |

ES3 | imsmetn | 49.963 | 0.105 |

HU1 | imsmetn | 128.311 | -0.407 |

HU2 | imsmetn | 63.555 | -0.246 |

HU3 | imsmetn | 149.543 | -0.406 |

NO2 | imsmetn | 78.555 | -0.241 |

NO3 | imsmetn | 29.665 | -0.137 |

PT1 | imsmetn | 108.786 | 0.163 |

PT2 | imsmetn | 115.86 | 0.17 |

PT3 | imsmetn | 241.82 | 0.209 |

PT3 | imdfetn | 135.839 | -0.122 |

SI2 | imsmetn | 22.375 | 0.108 |

Modification index larger than 20 and a parameter change larger than 0.10.

In these specific cases, the assumption of equal factor loadings is untenable. If we were to give up the equality constraint on the factor loading of IMSMET in group AT1 (Austria at time point 1), for example, the chi-square value would drop by 33.94 units. The freely estimated factor loading would be 0.107 higher than now estimated (this is the expected parameter change). Specifically, this means that, in this group, attitudes towards immigrants from the same ethnic group are more closely connected to anti-immigration attitudes in general than in other groups.

It is remarkable that almost all violations refer to the same item, namely IMSMETN. This indicates that factor loadings for the item on immigrants from the same group vary considerably across groups, while the meaning of the other two items is more cross-culturally robust. In the very few cases where a different items is involved (groups DK2, DK3 and PT3), the factor loading for IMSMETN is violated as well, and to a much greater extent. Most probably, freeing the loading for IMSMETN would also solve the problem for the other item.

Now we take a closer look at the modification indices for the measurement intercepts. Click ‘intercepts’ in the upper left corner of the output window and scroll through the groups.

### Question

Do you see violated equality constraints (i.e. a modification index larger than 20 and a parameter change larger than 0.10 in absolute value)? If so, in which groups and for which items?

SolutionWe find even more violated constraints on the intercepts than on the factor loadings:

Group | Parameter | Modification Index | Parameter change |
---|---|---|---|

AT1 | imsmetn | 104.657 | 0.119 |

AT3 | imsmetn | 103.782 | -0.129 |

CH2 | imsmetn | 160.483 | -0.166 |

CH3 | imsmetn | 193.101 | -0.217 |

DE2 | imsmetn | 112.686 | -0.13 |

DE2 | impcntr | 74.386 | 0.1 |

DE3 | imsmetn | 131.481 | -0.138 |

DK1 | imsmetn | 147.913 | -0.197 |

DK2 | imsmetn | 287.81 | -0.324 |

DK2 | impcntr | 60.796 | 0.119 |

DK3 | imsmetn | 346.252 | -0.363 |

DK3 | impcntr | 76.311 | 0.132 |

ES1 | imsmetn | 140.924 | 0.133 |

ES2 | imsmetn | 126.432 | 0.133 |

ES3 | imsmetn | 274.059 | 0.181 |

FI3 | impcntr | 98.745 | 0.125 |

HU1 | impcntr | 109.019 | 0.137 |

HU1 | imdfetn | 102.614 | 0.128 |

HU1 | imsmetn | 195.478 | -0.331 |

HU2 | impcntr | 135.029 | 0.199 |

HU2 | imdfetn | 54.169 | 0.112 |

HU2 | imsmetn | 123.575 | -0.275 |

HU3 | impcntr | 244.826 | 0.273 |

HU3 | imdfetn | 59.741 | 0.124 |

HU3 | imsmetn | 203.752 | -0.378 |

NL1 | imsmetn | 140.326 | 0.112 |

NO1 | impcntr | 91.435 | -0.105 |

NO2 | imsmetn | 65.157 | -0.123 |

NO3 | imsmetn | 104.006 | -0.15 |

PT1 | imsmetn | 262.235 | 0.189 |

PT2 | imsmetn | 242.332 | 0.185 |

PT3 | imsmetn | 446.51 | 0.232 |

SE1 | impcntr | 118.767 | -0.109 |

Modification index larger than 20 and a parameter change larger than 0.10.

In Austria at time point 1, setting the intercept of IMSMETN free would cause the chi-square value of the model to drop by 104,657. The freely estimated intercept would increase by 0.119.

The vast majority of the violations are related to IMSMETN here as well.