Differences in correlations

Given these results, we can expect the correspondence between the different measures for the same concepts to be rather low. Let us now look at what happens to the correlations between the measures of the different traits. Table 1.3 presents the correlations between the nine measures.

Table 1.3: Correlations between the nine variables of the MTMM experiment with respect to satisfaction with political outcomes obtained in the ESS Round 1 Pilot study for Great Britain

In Table 1.3, for example, the correlation between Methods 1 and 2 for the concept of satisfaction with democracy (Q3) was -0.669. For Methods 1 and 3 the correlation was -0.566, while it was 0.638 between Methods 2 and 3. Each observed variable was supposed to measure the same concept. So, we would expect to find correlations between these variables that are very close to 1 if there were no measurement errors. However, we see that these relationships are far from perfect (1.0). The conclusion is therefore clear that all three question forms contain errors.

These results also indicate a need for further investigation of the quality of the different measures, since the correlations between the three requests Q1 to Q3 are very different for the different methods. For the first method, the correlations vary between 0.373 and 0.552; for the second method, between 0.612 and 0.693; and, for the third method, between 0.514 and 0.558.

Since the correlations between the different forms of the questions for the same concept are not 1, we know that there are errors in these questions and we have also seen that these differences lead to large differences in frequency distributions and correlations. This shows that it is important to know where these errors come from and how they can be estimated. This is the topic of the next two chapters.

Exercise 1.1:

After the experiment in the pilot of the ESS, it was decided that the 11-point scale is better and that form has been chosen for the main questionnaire. However, in ESS Round 1,1 a second experiment was conducted with different forms of the same questions. The experiment consisted of nine questions measuring the three concepts (or traits) defined in this chapter (i.e. satisfaction with the economy, government and democracy). The first method was presented in the ESS Round 1 Main Questionnaire for all countries, while the other two methods were presented in the ESS Round 1 Supplementary Questionnaires.

Questions and scales

In order to carry out the analysis of the ESS Round 1 data for Great Britain, MTMM data can be downloaded from the links below:

Download data in SPSS format
Download data in Stata format

Answer the following questions:

  1. Create a table for the correlations of the questions (i.e. satisfaction with the economy, government and democracy) using Method 1, Method 2 and Method 3. You will be able to find the different method variables in the dataset under the names specified in the following table.

Solution for SPSS users

In this illustration, we will explain how to obtain the correlations of the variables in this chapter in SPSS.2 The SPSS syntax is provided below step by step.

  1. First, open the dataset in SPSS:

    GET FILE='C:\...\ESSround1MTMMs.sav'.
    DATASET NAME DataSet1 WINDOW=FRONT.
  2. Select the cases under study. They concern the whole British population. Therefore, in SPSS under the heading Data, select ‘Select Cases…’. To limit the analysis to Great Britain, choose ‘If condition is satisfied’, select the variable ‘Country’ and insert the following notation: cntry = ‘GB’.

    USE ALL.
    COMPUTE filter_$=(cntry = "GB").
    VARIABLE LABELS filter_$ 'cntry = "GB" (FILTER)'.
    VALUE LABELS filter_$ 0 'Not Selected' 1 'Selected'.
    FORMATS filter_$ (f1.0).
    FILTER BY filter_$.
    EXECUTE.
  3. Under the heading Data, you can weight the cases using design weights. Select ‘Weight cases’ and weight the cases by the variable ‘Design weight [dweight]’.

    WEIGHT BY dweight.
  4. Maintaining the above limitation to the British population and the design weights, the correlation matrix can be obtained from the SPSS heading Analyze under the ‘Correlate – Bivariate’ option. In this screen, you again select the nine variables used in this analysis.

    CORRELATIONS
    /VARIABLES=stfeco stfgov stfdem test7 test8 test9 test25 test26 test27
    /PRINT=TWOTAIL NOSIG
    /MISSING=PAIRWISE.

The results3 are summarized in the following table:

All correlations are significant at 1% significance level (*). Methods 2 and 3 were provided in the supplementary questionnaire of ESS Round 1 after the main questionnaire was answered. The sample was divided into two sub-groups and each group received one supplementary questionnaire (Method 2 or Method 3). For this reason, there are no correlations between the variables of Method 2 and Method 3.

Solution for Stata users

In this illustration, we will explain how to obtain the correlations of the variables in this chapter in Stata.4 The Stata syntax is provided below step by step.

  1. First, open the dataset in Stata:

    use "C:\...\ESSround1MTMMs.dta", clear
  2. Carrying out some tabulations of our variables of interest before asking for the correlation, we see that the variables Satdem [stfdem], Free [fairelecc], Critic [oppcrgvc], Equal [cttresac], LRplace [lrscale] and Inc [inctnta] have Refusal and Don’t know values that should be assigned to system missing. This can be done using the command ‘mvdecode’.

    mvdecode stfdem fairelcc oppcrgvc cttresac lrscale hinctnta, mv(99)
    mvdecode stfdem fairelcc oppcrgvc cttresac lrscale hinctnta, mv(88)
    mvdecode stfdem fairelcc oppcrgvc cttresac lrscale hinctnta, mv(77)
  3. Select the cases under study. They concern the whole British population. Therefore, in Stata we can use the command ‘keep if’ and indicate that we will keep all observations that for the variable ‘Country (cntry)’ have the value ‘GB’.

    keep if cntry=="GB"
  4. Maintaining the above limitation to the British population, the correlation matrix can be obtained from the Stata using the command ‘corr’. Using this command, again select the nine variables used in this analysis. Here, the design weights have been applied using the command ‘aweight’. Besides, the option ‘star(.01)’ will print an star next to each 1% significant coefficient.

    pwcorr stfeco stfgov stfdem test7 test8 test9 test25 test26 test27 [aweight=dweight], star(.01)

The results5 are summarized in the following table:

All correlations are significant at 1% significance level (*). Methods 2 and 3 were provided in the supplementary questionnaire of ESS Round 1 after the main questionnaire was answered. The sample was divided into two sub-groups and each group received one supplementary questionnaire (Method 2 or Method 3). For this reason, there are no correlations between the variables of Method 2 and Method 3.

  1. Do these results indicate that there are measurement errors in the responses to these questions?

Solution

Looking at the correlation matrix obtained, we can easily see that these relationships are quite far from perfect (1.00) for the questions measuring the same variable by a different method. More specifically, the correction between the questions about satisfaction with the economy for Method 1 (Stfeco) and Method 2 (Test7) is 0.535, while for Method 1 (Stfeco) and Method 3 (Test25) it is 0.603. For the questions about satisfaction with the government, the correlation is 0.674 between Method 1 (Stfgov) and Method 2 (Test8) and 0.766 between Method 1 (Stfeco) and Method 3 (Test26). In the same way, we can see that, for the questions about satisfaction with democracy, the correlation is 0.528 between Method 1 (Stfdem) and Method 2 (Test9) and 0.646 between Method 1 (Stfdem) and Method 3 (Test27).

Within methods, the differences in correlations are not that large. The correlations between the three questions for Methods 1 and 2 do not differ much, but Method 3 produces quite different results, i.e. higher correlations.

Given these results, it is clear that these questions contain measurement errors because the correlations between the questions measuring the same variables are very far from 1. Moreover, the correlations for the three variables across methods are not the same, which also indicates that there must be errors in one or more methods.

Go to next chapter >>

Footnotes