SQP 2.0 application

In order to continue with the example introduced in this module, we are going to illustrate the first step before the analysis with corrections for measurement errors is run. This consists of obtaining the quality predictions provided by the Survey Quality Predictor (SQP) 2.0. Thus, for this illustration of the applicability of SQP 2.0, we are going to focus on the variables summarized in Table 3.1.

Table 3.1: ESS Round 6 variables included in the analysis of the evaluation of democracy

As indicated above, the codes of the question characteristics and the quality predictions are already available in SQP 2.0 for many questions. For other questions, the user has to code the characteristics of the questions to obtain the quality predictions. In this specific case, the characteristics of the questions in the English (United Kingdom) version have already been coded. However, in order to obtain the predictions for questions in other languages, the questions still have to be introduced and coded.

For the purpose of this illustration, the coding procedure started with two coders coding the characteristics of the questions independently. They completed one coding each and the differences between the codes were discussed. The final coding decisions were applied in SQP 2.0 and the final coding now appears as authorized. It is important to remember that the authorized codes are reliable because they have been done under the supervision of SQP staff.

For the sake of simplicity, we will not go through all the coding decisions taken, since this would require a rather lengthy discussion that could easily distract the reader from the main purpose of this module: how to correct for measurement errors. However, if you are interested in reproducing the analysis in another language or other variables, we refer to Appendix 1. This appendix is a good introduction to the use of SQP and it also provides a step-by-step detailed explanation of all coding decisions taken for the questions above. If you understand the decisions taken, this procedure can be replicated in many other languages or variables.

For the British version of these questions, the procedure for obtaining the quality predictions is relatively simple since the characteristics have already been coded. For those who are interested, a simple illustration of how we can obtain the available quality predictions from SQP is presented in the following link.

Viewing quality predictions in SQP

Step 1: Access SQP 2.0 through the link: sqp.upf.edu. If you are already a user, log in. But if you are a new user, please register and create an account, which is free of charge.

Step 2: Once in the Home page of SQP, you should click the link ‘All Available Questions’.

Next, the list of all available questions in SQP will appear. Because of the huge amount of questions currently available (more than 65,000), it would not be easy to search for a specific question without filtering its characteristics.

Step 3: To find the questions of interest, you can use the filter options on the left-hand side of the screen to fill in the details of the questions. These details are provided for the variables in our example in Table 3.2.

Table 3.2: Information about the variables to be studied using SQP 2.0

For example, to search for question B23 (Satdem), we will have to select the following information from the browser lists: in ‘All Studies’ select ‘ESS Round 6’, in ‘All Languages’ select ‘English’ and in ‘All Countries’ choose ‘United Kingdom’. These filters will provide you with the complete list of ESS Round 6 questions in the UK. In addition, we can also specify the name of the question, ‘B23’, using the ‘Containing text’ option. If you have followed all these steps, the following screen with our unique question of interest should appear:

Step 4: If you click on the question, a smaller screen pop-up will appear with information about this question. This information includes the question and answer texts and the quality predictions obtained by other users.

In the section on the right of the quality information in Screen 3.4, we see two available links. The first one refers to the authorized quality prediction obtained by the coding steps defined in Appendix 1. Furthermore, the second link gives the user an opportunity to code the question him/herself. For the purpose of this illustration, we will continue by viewing the prediction detail obtained by the authorized prediction. Thus, to continue, click the link: ‘View prediction detail’, which takes you to Screen 3.5.

This screen provides the quality information obtained for this particular question and particular coding. It should be clear that, if different coding decisions were made, the values obtained would also differ. For more precise information about the quality coefficients, you should click the link ‘View Quality Coefficients’. This link will lead the user to the following screen:

Screens 3.5 and 3.6 provide information about the quality criteria for the question of interest, in this case B23. Screen 3.5 provides predictions of the reliability, validity and quality, while Screen 3.6 provides the square root of these values, i.e. the quality coefficients. Furthermore, the other columns in Screen 3.6 provide information about the uncertainty of these predictions.

Using SQP 2.0, we are able to obtain the predicted values of reliability (r2), validity (v2) and quality (q2), and also the coefficients (r), (v) and (q), for all questions. These latter coefficients will be used to calculate the common method variance (cmv).1

The results for all questions used in our illustration are presented in Table 3.3. Note that the information in the first row of this table can be obtained from the illustration presented above (see Screens 3.5 and 3.6 in "Viewing quality predictions in SQP").

Table 3.3: The SQP quality predictions for the questions under study of ESS Round 6 for Great Britain

Table 3.3 shows that the quality of the questions does not vary much, although we can also see that the quality deviates quite far from 1. This means that the size of the errors in these questions is considerable. In the next chapter, we will see that these deviations will have a considerable effect on the size of the correlations if we correct for measurement errors.

To summarize, this chapter has shown that it is now relatively simple to obtain estimates of the quality of the questions in a study. This means that correction for measurement error is also much simpler than it was before.

Exercise 3.1:

In ESS Round 3, four variables were expected to provide an explanation of opinions about immigration by people from outside Europe. Questions about these variables were asked in all countries using the following formulations.2 Before B37, people were asked to give their opinions about immigration by people of the same race or ethnic group and by people from other European countries. After that, question B37 was asked as shown below.

Questions on immigration (Open in new window: B37 - B38 - B39 - B40)

Based on these questions, access the program Survey Quality Predictor (SQP) and add the missing spaces about the quality predictions of these questions in the table below. For this particular illustration, we have chosen the Netherlands case for reasons that will become clear later on.3 The table provides the information necessary to find the questions in SQP, which are already coded and authorized.

Input for exercise*
Question r2 v2 q2 m**
B31 (Allow) 0.840 0.763
B38 (Economy) 0.875 0.354
B39 (Culture) 0.777 0.641
B40 (Better) 0.878 0.349

*SQP Study = ESS Round 3, SQP Country = Netherlands, SQP Language = Dutch
**m is defined as mi = √(1-vj2).


In the following, a description of the steps necessary to obtain the results in SQP is presented:

  1. Access sqp.upf.edu, and register if you are not already a user.
  2. On the Home page of SQP, follow the link: ‘View all questions that are currently available >>’. This link will take you to the list of all available questions in SQP.
  3. On the left side of the screen, users can easily search for the desired questions by filtering the known details (Study, Country and Language). Furthermore, in the ‘Containing text’ option, you can write the name of the question of interest (e.g. B37).
  4. Introduce the details. For question B37, select from the browser list ‘All Studies’ the study ‘ESS Round 3’; from the browser list ‘All Languages’, select the language ‘Dutch; from the browser list ‘All Countries’, select the country ‘Netherlands’, and, in the ‘Containing text’ option, enter the question name ‘B37’.
  5. The question list is reduced to just one question with the specified characteristics. Select the question and you will be able to see the question text and the authorized prediction (i.e. the prediction made by trained coders under the supervision of the SQP team). Click ‘View Prediction detail’ to obtain the quality values of reliability, validity and quality.
  6. From the SQP quality predictions screen, you can get the necessary information to fill in the table. Specifically, for this question B37, we get 0.840, 0.908 and 0.763 for the reliability, validity and quality values, respectively.
  7. Repeat the exact same procedure (steps 3 to 6) to obtain the coefficients for the other questions. The only information that should be changed each time is the question name in the ‘Containing text’ option.
  8. The method effect (m) coefficient can be computed as: mi = √(1-v2).
    mB37 = √(1-0.908) = 0.303
    mB38 = √(1-0.875) = 0.354
    mB39 = √(1-0.825) = 0.418
    mB40 = √(1-0.878) = 0.349
Question r2 v2 q2 M
B31 (Allow) 0.840 0.908 0.763 0.303
B38 (Economy) 0.802 0.875 0.702 0.354
B39 (Culture) 0.777 0.825 0.641 0.418
B40 (Better) 0.728 0.878 0.639 0.349

*SQP Study = ESS Round 3, SQP Country = Netherlands, SQP Language = Dutch

Go to next chapter >>