Exercise 3.3: The statistical significance of attitude trends
Up until now, we have considered all differences between time points as meaningful. However, the observed evolution may be caused by sampling fluctuations. To be more confident that the mean differences are not just a matter of chance, we should perform statistical significance testing. Test every country separately to check whether the means at the three time points vary significantly. Use Analysis Of Variance (ANOVA) for this purpose (also known as an F-test for mean differences between groups). HINT: use the SPLIT FILE statement to perform the analysis country-by-country in an efficient manner.
*Weight data and test every country separately to check whether the means at the three time points vary significantly.
For which countries are the differences between time points statistically significant?
|Country||Groups||Sum of Squares||df||Mean Square||F||Sig.|
|United Kingdom||Between Groups||100.687||2||50.343||9.975||.000|
Weighted by design weight.
Analysis Of Variance (ANOVA) can be used to test whether the mean of a certain variable (here REJECT) varies across categories of a second variable (here: ESSROUND, indicating the three time points). Here, we performed a separate ANOVA for each of the 17 countries. The test basically compares the amount of variation in REJECT between time points with the amount of variation within time points. Logically, the larger the differences between time points, the more convincing is the evidence that attitude changes have taken place. The F-test can be used to test whether differences between time points are statistically significant. The null hypothesis of this test is that the means are equal at all three time points. If the p-value of the test statistic is lower than .05, this null hypothesis can be rejected. In the latter case, we can conclude with 95 per cent certainty that the observed attitude changes are not due to chance fluctuations.
The SPSS output from the ANOVA shows that, in 13 out of 17 countries, attitude evolution is statistically significant at the .05 level. Only in Finland, France, Ireland and Slovenia were the mean differences between time points too small to be conclusive.
In fact, it is not very surprising that we find many attitude changes to be statistically significant. Significance depends not only on the size of the differences, but also on the sample size. Here, we are clearly dealing with very large datasets: our analyses include information about more than 80,000 respondents. As a result, even small attitude changes become significant. We should not, therefore, rely blindly on significance tests, but also look at whether the mean differences are substantially large. Remember that the strongest attitude change was found in Poland. There, the average score on REJECT dropped by 0.85 points. Given that REJECT has a minimum value of 3 and a maximum of 12 (REJECT is the sum of three items measured on a 1 to 4 scale), this is a very substantial change indeed. We also find substantial mean differences greater than 0.30 in Austria, Germany, Spain, Hungary and Poland.