Exercise 3.3: The statistical significance of attitude trends

Up until now, we have considered all differences between time points as meaningful. However, the observed evolution may be caused by sampling fluctuations. To be more confident that the mean differences are not just a matter of chance, we should perform statistical significance testing. Test every country separately to check whether the means at the three time points vary significantly. Use Analysis Of Variance (ANOVA) for this purpose (also known as an F-test for mean differences between groups). HINT: use the SPLIT FILE statement to perform the analysis country-by-country in an efficient manner.

*Weight data and test every country separately to check whether the means at the three time points vary significantly.

WEIGHT by dweight.

SORT CASES by cntry.
SPLIT FILE by cntry.
MEANS reject by essround
/statistics=anova.
SPLIT FILE off.
WEIGHT off.
SAVE outfile = 'C:\ESS123_immig.sav'

Question

For which countries are the differences between time points statistically significant?

Table 3.2. Analysis of variance: Do the means at the three time points vary significantly?
Country Groups Sum of Squares df Mean Square F Sig.
Austria Between Groups 289.064 2 144.532 30.720 .000
Within Groups 26272.060 5584 4.705
Total 26561.124 5586
Belgium Between Groups 71.057 2 35.528 6.826 .001
Within Groups 24915.424 4787 5.205
Total 24986.481 4789
Switzerland Between Groups 30.790 2 15.395 4.300 .014
Within Groups 16528.847 4617 3.580
Total 16559.637 4619
Germany Between Groups 416.271 2 208.135 41.634 .000
Within Groups 37494.135 7500 4.999
Total 37910.405 7502
Denmark Between Groups 52.891 2 26.445 6.672 .001
Within Groups 15962.638 4027 3.964
Total 16015.529 4029
Spain Between Groups 220.836 2 110.418 17.584 .000
Within Groups 27811.598 4429 6.279
Total 28032.434 4431
Finland Between Groups 9.876 2 4.938 1.229 .293
Within Groups 22412.782 5579 4.017
Total 22422.659 5581
France Between Groups 1.528 2 .764 .162 .851
Within Groups 21198.352 4485 4.726
Total 21199.880 4487
United Kingdom Between Groups 100.687 2 50.343 9.975 .000
Within Groups 27238.776 5397 5.047
Total 27339.463 5399
Hungary Between Groups 98.259 2 49.130 11.283 .000
Within Groups 16812.166 3861 4.354
Total 16910.425 3863
Ireland Between Groups 2.258 2 1.129 .243 .784
Within Groups 24017.504 5171 4.645
Total 24019.762 5173
Netherlands Between Groups 62.020 2 31.010 6.822 .001
Within Groups 24723.600 5439 4.546
Total 24785.620 5441
Norway Between Groups 28.794 2 14.397 3.710 .025
Within Groups 19579.635 5046 3.880
Total 19608.430 5048
Poland Between Groups 619.773 2 309.886 61.726 .000
Within Groups 24921.187 4964 5.020
Total 25540.959 4966
Portugal Between Groups 76.289 2 38.144 6.524 .001
Within Groups 27860.564 4765 5.847
Total 27936.853 4767
Sweden Between Groups 41.306 2 20.653 5.137 .006
Within Groups 20010.046 4977 4.021
Total 20051.352 4979
Slovenia Between Groups 4.282 2 2.141 .431 .650
Within Groups 18057.961 3632 4.972
Total 18062.244 3634

Weighted by design weight.

Analysis Of Variance (ANOVA) can be used to test whether the mean of a certain variable (here REJECT) varies across categories of a second variable (here: ESSROUND, indicating the three time points). Here, we performed a separate ANOVA for each of the 17 countries. The test basically compares the amount of variation in REJECT between time points with the amount of variation within time points. Logically, the larger the differences between time points, the more convincing is the evidence that attitude changes have taken place. The F-test can be used to test whether differences between time points are statistically significant. The null hypothesis of this test is that the means are equal at all three time points. If the p-value of the test statistic is lower than .05, this null hypothesis can be rejected. In the latter case, we can conclude with 95 per cent certainty that the observed attitude changes are not due to chance fluctuations.

The SPSS output from the ANOVA shows that, in 13 out of 17 countries, attitude evolution is statistically significant at the .05 level. Only in Finland, France, Ireland and Slovenia were the mean differences between time points too small to be conclusive.

In fact, it is not very surprising that we find many attitude changes to be statistically significant. Significance depends not only on the size of the differences, but also on the sample size. Here, we are clearly dealing with very large datasets: our analyses include information about more than 80,000 respondents. As a result, even small attitude changes become significant. We should not, therefore, rely blindly on significance tests, but also look at whether the mean differences are substantially large. Remember that the strongest attitude change was found in Poland. There, the average score on REJECT dropped by 0.85 points. Given that REJECT has a minimum value of 3 and a maximum of 12 (REJECT is the sum of three items measured on a 1 to 4 scale), this is a very substantial change indeed. We also find substantial mean differences greater than 0.30 in Austria, Germany, Spain, Hungary and Poland.

Go to next chapter >>