Exercise 2.1, step 5: Interpretation of the output

Now comes the most important step of the analysis: the interpretation of the output. The output can be accessed by clicking the ‘View text’ button.

 

Figure 2.12: 'View text'

The estimation has produced a massive amount of output. The AMOS ‘Output window’ allows you to navigate through this output in a very efficient way by selecting different parts of the output, different groups and different models.

 

Figure 2.13: Output window

We start by inspecting the fit of the different models. Click ‘Model fit’, and then ‘CMIN’. The ‘CMIN’ heading contains the chi-square values and the number of degrees of freedom for the various models. Remember that we are only interested in three models: unconstrained (configural equivalence), measurement weights (metric equivalence) and measurement intercepts (scalar equivalence).

 

Figure 2.14: Model fit on 'CMIN'

Question

What do these chi-square values tell you about the fit of the models?

Solution

The unconstrained model has a chi-square value of 0. This is not surprising - models with only three indicators and without additional constraints are just-identified (this means that they have 0 degrees of freedom). Fit indices for this model are therefore not very informative. The ‘measurement weights’ model has a chi-square value of 1.573,024 for 100 degrees of freedom; the measurement incepts model has a chi-square value of 15.536,361 for 250 degrees of freedom. The higher the chi-square value, the larger the discrepancy between our model and the observed data. The p-values for both models are 0. This indicates that it is very unlikely that the model will give an adequate description of the data. In other words, the chi-square tests indicate bad model fit and reject metric as well as scalar equivalence. However, chi-square tests are known to be very sensitive to large sample sizes. If the sample size is sufficiently large, even negligible misspecifications in the model may lead to the rejection of the whole model. Because we are analysing a massive sample size here (over 80,000 respondents!), it is better not to pay too much attention to these chi-square tests.

To avoid the problems of these chi-square tests, alternative indices of model fit have been developed. Root Mean Squared Error of Approximation (RMSEA) and Comparative Fit Index (CFI) are two very informative measures of how close the model corresponds with the data. Click on ‘Baseline Comparisons’ and ‘RMSEA’ to view this alternative measure of model fit.

 

Figure 2.15: Model fit on 'Baseline Comparisons'

 

Figure 2.16: Model fit on 'RMSEA'

Question

What do you conclude based on the CFI and RMSEA values?

Solution

As a rule of thumb, RMSEA values lower than 0.05 are considered to indicate acceptable model fit. Judging by RMSEA, the ‘measurement weights’ and the ‘measurements intercepts’ models fit the data reasonably well. Generally, it is assumed that models with CFI values larger than 0.90 are acceptable. Thus, the CFI-criterion also points in the direction of reasonable model fit, although the CFI of the ‘measurement intercepts’ model only marginally exceeds 0.90.

Overall fit indices such as RMSEA and CFI thus seem to provide some evidence that the REJECT-scale is comparable across countries and time points. However, to test measurement equivalence, we should not rely completely on RMSEA and CFI. RMSEA and CFI are measures of overall model fit. They summarise the goodness-of-fit of a complete model in a single number. The model could, for example, contain a severe misspecification in one of the groups or for one specific parameter, but still have a reasonable overall fit. Therefore, we should also check that no misfit is present in various parts of the model separately. In our case, a thorough test of measurement equivalence requires that we look at possible misfit due to the constrained factor loadings and intercepts. So-called ‘modification indices’ are a very useful tool for inspecting for local misfit. For all constrained parameters in the model, AMOS calculates a modification index. Modification indices indicate how much the chi-square value of a model would drop if the parameter were free instead of constrained (in other words, by how much the model fit would improve). Modification indices are in fact chi-square tests for individual equality constraints: high values indicate that the respective parameter constraint is ‘wrong’. However, since chi-square values are known to be very sensitive to large sample sizes (cf. supra), only really high values (larger than 20, say) should be taken as serious evidence of misfit. Because the chi-square values can be misleading, it is advisable to look at the expected parameter change as well. The parameter change indicates how much a parameter would change if the equality constraint were removed. Obviously, we are only interested in parameter changes that are substantial (larger than 0.10 in absolute value, say).

To summarise, a conclusive test of measurement equivalence can be performed by looking at modification indices and expected parameter changes for the factor loadings and measurement intercepts. We start by looking at the equality constraints for the factor loadings. Click ‘modification indices’ and then ‘measurement weights (as mentioned above, these are the factor loadings). Next, select the ‘Measurement intercepts’ model in the bottom left corner of the output window, (this is the model with the constrained factor loadings and intercepts). Now you see modification indices and parameter changes for the first group (AT1). Not all the modification indices presented refer to factor loadings - you should only pay attention to relations between items and the latent variable F1 (these modification indices are indicated by a statement with the following structure: ‘item name <--- F1’). Modification indices for the other groups can be examined by scrolling through the groups in the left-hand column.

 

  1. Select 'Modification indices - Regression weights'
  2. Select 'Measurement intercepts' model
  3. Scroll through the groups

 

Figure 2.17: Modification indices and parameter changes for the factor loadings in group AT1

Question

Inspect modification indices and parameter changes for the factor loadings in each of the 51 groups. Do you see violated equality constraints (i.e. a modification index larger than 20 and a parameter change larger than 0.10 in absolute value)? If so, in which groups and for which items?

Solution

Quite a few equality constraints on the factor loadings appear to be violated. An overview of violations can be found in the table below:

Table 2.1. Violated constraints on the factor loadings
Group Parameter Modification Index Parameter change
AT1 imsmetn 33.94 0.107
CH2 imsmetn 23.343 -0.112
DK1 imsmetn 47.884 -0.2
DK2 imsmetn 112.224 -0.357
DK2 imdfetn 22.905 0.108
DK3 imsmetn 74.691 -0.309
DK3 imdfetn 32.185 0.131
ES1 imsmetn 45.221 0.109
ES3 imsmetn 49.963 0.105
HU1 imsmetn 128.311 -0.407
HU2 imsmetn 63.555 -0.246
HU3 imsmetn 149.543 -0.406
NO2 imsmetn 78.555 -0.241
NO3 imsmetn 29.665 -0.137
PT1 imsmetn 108.786 0.163
PT2 imsmetn 115.86 0.17
PT3 imsmetn 241.82 0.209
PT3 imdfetn 135.839 -0.122
SI2 imsmetn 22.375 0.108

Modification index larger than 20 and a parameter change larger than 0.10.

In these specific cases, the assumption of equal factor loadings is untenable. If we were to give up the equality constraint on the factor loading of IMSMET in group AT1 (Austria at time point 1), for example, the chi-square value would drop by 33.94 units. The freely estimated factor loading would be 0.107 higher than now estimated (this is the expected parameter change). Specifically, this means that, in this group, attitudes towards immigrants from the same ethnic group are more closely connected to anti-immigration attitudes in general than in other groups.

It is remarkable that almost all violations refer to the same item, namely IMSMETN. This indicates that factor loadings for the item on immigrants from the same group vary considerably across groups, while the meaning of the other two items is more cross-culturally robust. In the very few cases where a different items is involved (groups DK2, DK3 and PT3), the factor loading for IMSMETN is violated as well, and to a much greater extent. Most probably, freeing the loading for IMSMETN would also solve the problem for the other item.

Now we take a closer look at the modification indices for the measurement intercepts. Click ‘intercepts’ in the upper left corner of the output window and scroll through the groups.

Question

Do you see violated equality constraints (i.e. a modification index larger than 20 and a parameter change larger than 0.10 in absolute value)? If so, in which groups and for which items?

Solution

We find even more violated constraints on the intercepts than on the factor loadings:

Table 2.2. Violated constraints on the intercepts
Group Parameter Modification Index Parameter change
AT1 imsmetn 104.657 0.119
AT3 imsmetn 103.782 -0.129
CH2 imsmetn 160.483 -0.166
CH3 imsmetn 193.101 -0.217
DE2 imsmetn 112.686 -0.13
DE2 impcntr 74.386 0.1
DE3 imsmetn 131.481 -0.138
DK1 imsmetn 147.913 -0.197
DK2 imsmetn 287.81 -0.324
DK2 impcntr 60.796 0.119
DK3 imsmetn 346.252 -0.363
DK3 impcntr 76.311 0.132
ES1 imsmetn 140.924 0.133
ES2 imsmetn 126.432 0.133
ES3 imsmetn 274.059 0.181
FI3 impcntr 98.745 0.125
HU1 impcntr 109.019 0.137
HU1 imdfetn 102.614 0.128
HU1 imsmetn 195.478 -0.331
HU2 impcntr 135.029 0.199
HU2 imdfetn 54.169 0.112
HU2 imsmetn 123.575 -0.275
HU3 impcntr 244.826 0.273
HU3 imdfetn 59.741 0.124
HU3 imsmetn 203.752 -0.378
NL1 imsmetn 140.326 0.112
NO1 impcntr 91.435 -0.105
NO2 imsmetn 65.157 -0.123
NO3 imsmetn 104.006 -0.15
PT1 imsmetn 262.235 0.189
PT2 imsmetn 242.332 0.185
PT3 imsmetn 446.51 0.232
SE1 impcntr 118.767 -0.109

Modification index larger than 20 and a parameter change larger than 0.10.

In Austria at time point 1, setting the intercept of IMSMETN free would cause the chi-square value of the model to drop by 104,657. The freely estimated intercept would increase by 0.119.

The vast majority of the violations are related to IMSMETN here as well.

Go to next page >>