All pages

The estimation of the quality of survey questions

There are many different procedures for estimating the quality of questions and of measures for complex concepts. The best known is perhaps the test-retest design [Lor68] for estimating the reliability of questions. An adjustment of this approach was the Quasi-simplex model [Hei69], [Wil70] used by [Alw91] and [Alw07]. The Multitrait - Multimethod (MTMM) design was suggested in order to take into account the effects of the method used [Cam59]. It was further developed by Andrews [And84] and others for survey questions. For concepts with multiple indicators, different procedures have been developed based on latent variable models such as factor analysis [Law71], [Har76] and latent class analysis [Hag88], [Ver03], [Bie11]. Furthermore, scaling methods have been developed, such as the Thurstone scale, Likert scale etc. [Tor58], the Gutmann scale and Mokken scale [Mok71], the Unfolding scale [Sch97], Rasch scale [Ras60] and Item Response theory [Ham91]. For the advantages and disadvantages of these different procedures, we refer to this literature.

All these procedures require at least two questions to estimate the quality of each concept. That means that the number of questions has to be at least twice the number of concepts one wishes to take into account in the analysis. As a result, these procedures lead to rather costly and time-consuming research involving rather complex procedures. Besides, all these procedures provide estimates of the quality of specific questions or concepts for the formulation of specific questions used in a specific questionnaire and context. Thus, generalization is not easily possible.

This means that a lot of research has to be done before the final data collection in order to correct for measurement errors in all variables in the study. This is so much work that it is only seldom done. So, the question is whether there is a procedure that is less time-consuming and expensive for estimating the quality of survey questions and of composite scores for concepts with multiple indicators.

From the very start of the European Social Survey (ESS), Saris has emphasized that the measures will contain errors and that, without correction for these errors, the results will be questionable and incomparable across countries. Therefore, since the beginning of 2002, each survey in the ESS has contained four to six MTMM experiments to evaluate the quality of the questions.1 These MTMM experiments were carried out in most countries and all rounds. An example of such an experiment was presented in the first chapter.

In the normal MTMM experiment suggested by [Cam59], the respondent has to provide responses to three different questions (i.e. traits) measured using three different methods [And84]. Because people had to answer the same question approximately three times, we might expect memory effects. In order to cope with the memory effects in the MTMM experiments, it has been suggested by [Sar04] that the sample can be randomly split into different subgroups and the same question asked only twice in each group. This design was named the Split - Ballot Multitrait - Multimethod (SB-MTMM) design. [Sar04] also showed that this design enables estimation of the reliability and validity (complement of the method effect) and the quality of each question.2 In recent years, all experiments in the ESS rounds have been analysed using the SB-MTMM procedure. Consequently, after the first three ESS rounds, more than 250 SB-MTMM experiments have been conducted in more than 20 countries (languages), including approximately 2,700 questions. Thus, because of the results obtained after the first three rounds of the ESS, together with the results obtained by previous and simultaneous MTMM analysis done by other research agencies, the reliability, validity and quality of 3,726 questions is now known. However, this information is not enough because, at the same time in the ESS, more than 62,000 questions were asked about values, norms, policy preferences, feelings etc. the quality of which was not analysed. Thus, a different approach was required.

Page 1

From estimation to prediction

The new idea was to code the characteristics1 of the questions involved in the experiments and, with this knowledge, to develop an algorithm to predict the quality of the questions [Sar07]. If that prediction proved successful, the same algorithm could be used to predict the quality of any other question not necessarily involved in the experiments. Work on this new approach has been subsidized by the European Commission for Infrastructure research and has led to the development of the program Survey Quality Predictor (SQP). It contains the quality information relating to the questions involved in the MTMM experiments, but can also be used to predict the quality of questions from other studies. For the complete report about the development of SQP 2.0, we refer to [Sar14] and [Sar11]. Here, we only mention the basic steps that were introduced in this process.

The first step was to use the characteristics and the context of the questions involved in MTMM experiments as predictors of their quality. Therefore, a program was developed to code the characteristics of the questions that were involved in the MTMM experiments. For details of these characteristics, we refer to [Sar11]. People who spoke the different languages involved in the experiments and who were able to understand English coded the questions. This was a very elaborate task, but the results were quite rewarding, as we will show below.

The next step was to choose a procedure to study the relationship between the question characteristics and the quality estimates for these questions. For this purpose, we did not choose the regression model used in the past [Sar07] but what is known as the ‘Random forest’ approach developed by [Bre01], because it was suggested to be the most efficient prediction procedure for this kind of problem.

It turned out that this procedure provided rather good predictions of the reliability and validity of our data. The explained variance (R2) for reliability was 0.65 and for validity 0.84. As a result, the prediction of the quality was rather good [Obe11].

Page 2

Survey Quality Predictor: SQP

Based on all this knowledge, this algorithm has been used to develop the computer program Survey Quality Predictor SQP 2.0,1 which generates predictions of the quality of survey questions [Obe11].

The Survey Quality Predictor 2.0 is a tool used to obtain predictions about the quality of survey questions. Nowadays, it provides a large database of more than 65,000 survey questions from many different questionnaires, in many different languages and about many different topics. The quality prediction is obtained by coding the characteristics in SQP. Currently, quality predictions are available for more than 10,000 survey questions. Besides, the Survey Quality Predictor is an open source tool under constant development, as the database is created with the collaboration of users. SQP users can be participants in the development of the program by adding new questions, coding their characteristics and obtaining quality predictions, or by coding the characteristics of any question already available in the database and obtaining quality predictions. It is important to note that, not only are the survey questions introduced by all users available to all SQP users, the predictions obtained by any user are also publicly available.

Thus, the Survey Quality Predictor has become a useful tool for questionnaire designers and researchers for gathering quality information about survey questions. This information is obtained without collecting new data. The only thing the researcher has to do is to code the characteristics of the questions. This is a major advantage compared with the procedures so far available for obtaining estimates of the quality of questions. In order to learn the standard process of the coding, we suggest taking a look at SQP 2.0.

The SQP team has made the ESS questionnaires available in the SQP database and organized the coding of the characteristics of the questions in the MTMM experiments in all the countries and languages participating since ESS Round 1. Thus, besides having more than 62,000 ESS questions from more than 20 different languages, an authorized quality prediction is available for more than 8,500 ESS questions. Since, on the other hand, other SQP users have coded questions as well, it is important to differentiate between user codes and authorized codes. The authorized codes can be trusted because they have been coded in different languages by native coders under the training and supervision of SQP members. Thus, if the question of interest is coded but not authorized, we suggest checking the coding before using the predictions.

To conclude, this means that researchers can now obtain, via SQP, a prediction of the quality of most ESS questions, but also of other new questions, without incurring costs other than the time required to introduce the question in the program and to code it. So, the program SQP makes it possible to obtain quality predictions for nearly all questions. A major problem for the researchers has thereby been solved, i.e. one only needs the estimates of the quality of all variables in the study in order to be able to correct for measurement errors in the analysis.

We say ‘nearly all questions’ because it is difficult to design MTMM experiments for background variables that also contain measurement errors, as has been shown by [Sch14] and [Alw07]. However, quality information about background variables can be obtained from [Alw07], who used quasi-simplex models. Combining the SQP predictions and the estimates of [Alw07] with respect to background variables, it is possible to obtain estimates of the quality of all questions used in survey research and it is therefore also possible to correct the correlations between all variables.

Furthermore, we should realize that SQP is not able to predict the quality of questions for countries in which no MTMM experiments have been done so far. At the moment, SQP can provide predictions for the following countries: Austria, Belgium, Czech Republic, Denmark, Estonia, Finland, France, Germany, Great Britain, Ireland, Netherlands, Norway, Poland, Portugal, Slovakia, Slovenia, Spain, Sweden, Switzerland, Ukraine and the United States.

Page 3

SQP 2.0 application

In order to continue with the example introduced in this module, we are going to illustrate the first step before the analysis with corrections for measurement errors is run. This consists of obtaining the quality predictions provided by the Survey Quality Predictor (SQP) 2.0. Thus, for this illustration of the applicability of SQP 2.0, we are going to focus on the variables summarized in Table 3.1.

Table 3.1: ESS Round 6 variables included in the analysis of the evaluation of democracy

As indicated above, the codes of the question characteristics and the quality predictions are already available in SQP 2.0 for many questions. For other questions, the user has to code the characteristics of the questions to obtain the quality predictions. In this specific case, the characteristics of the questions in the English (United Kingdom) version have already been coded. However, in order to obtain the predictions for questions in other languages, the questions still have to be introduced and coded.

For the purpose of this illustration, the coding procedure started with two coders coding the characteristics of the questions independently. They completed one coding each and the differences between the codes were discussed. The final coding decisions were applied in SQP 2.0 and the final coding now appears as authorized. It is important to remember that the authorized codes are reliable because they have been done under the supervision of SQP staff.

For the sake of simplicity, we will not go through all the coding decisions taken, since this would require a rather lengthy discussion that could easily distract the reader from the main purpose of this module: how to correct for measurement errors. However, if you are interested in reproducing the analysis in another language or other variables, we refer to Appendix 1. This appendix is a good introduction to the use of SQP and it also provides a step-by-step detailed explanation of all coding decisions taken for the questions above. If you understand the decisions taken, this procedure can be replicated in many other languages or variables.

For the British version of these questions, the procedure for obtaining the quality predictions is relatively simple since the characteristics have already been coded. For those who are interested, a simple illustration of how we can obtain the available quality predictions from SQP is presented in the following link.

Viewing quality predictions in SQP

Step 1: Access SQP 2.0 through the link: If you are already a user, log in. But if you are a new user, please register and create an account, which is free of charge.

Step 2: Once in the Home page of SQP, you should click the link ‘All Available Questions’.

Next, the list of all available questions in SQP will appear. Because of the huge amount of questions currently available (more than 65,000), it would not be easy to search for a specific question without filtering its characteristics.

Step 3: To find the questions of interest, you can use the filter options on the left-hand side of the screen to fill in the details of the questions. These details are provided for the variables in our example in Table 3.2.

Table 3.2: Information about the variables to be studied using SQP 2.0

For example, to search for question B23 (Satdem), we will have to select the following information from the browser lists: in ‘All Studies’ select ‘ESS Round 6’, in ‘All Languages’ select ‘English’ and in ‘All Countries’ choose ‘United Kingdom’. These filters will provide you with the complete list of ESS Round 6 questions in the UK. In addition, we can also specify the name of the question, ‘B23’, using the ‘Containing text’ option. If you have followed all these steps, the following screen with our unique question of interest should appear:

Step 4: If you click on the question, a smaller screen pop-up will appear with information about this question. This information includes the question and answer texts and the quality predictions obtained by other users.

In the section on the right of the quality information in Screen 3.4, we see two available links. The first one refers to the authorized quality prediction obtained by the coding steps defined in Appendix 1. Furthermore, the second link gives the user an opportunity to code the question him/herself. For the purpose of this illustration, we will continue by viewing the prediction detail obtained by the authorized prediction. Thus, to continue, click the link: ‘View prediction detail’, which takes you to Screen 3.5.

This screen provides the quality information obtained for this particular question and particular coding. It should be clear that, if different coding decisions were made, the values obtained would also differ. For more precise information about the quality coefficients, you should click the link ‘View Quality Coefficients’. This link will lead the user to the following screen:

Screens 3.5 and 3.6 provide information about the quality criteria for the question of interest, in this case B23. Screen 3.5 provides predictions of the reliability, validity and quality, while Screen 3.6 provides the square root of these values, i.e. the quality coefficients. Furthermore, the other columns in Screen 3.6 provide information about the uncertainty of these predictions.

Using SQP 2.0, we are able to obtain the predicted values of reliability (r2), validity (v2) and quality (q2), and also the coefficients (r), (v) and (q), for all questions. These latter coefficients will be used to calculate the common method variance (cmv).1

The results for all questions used in our illustration are presented in Table 3.3. Note that the information in the first row of this table can be obtained from the illustration presented above (see Screens 3.5 and 3.6 in "Viewing quality predictions in SQP").

Table 3.3: The SQP quality predictions for the questions under study of ESS Round 6 for Great Britain

Table 3.3 shows that the quality of the questions does not vary much, although we can also see that the quality deviates quite far from 1. This means that the size of the errors in these questions is considerable. In the next chapter, we will see that these deviations will have a considerable effect on the size of the correlations if we correct for measurement errors.

To summarize, this chapter has shown that it is now relatively simple to obtain estimates of the quality of the questions in a study. This means that correction for measurement error is also much simpler than it was before.

Exercise 3.1:

In ESS Round 3, four variables were expected to provide an explanation of opinions about immigration by people from outside Europe. Questions about these variables were asked in all countries using the following formulations.2 Before B37, people were asked to give their opinions about immigration by people of the same race or ethnic group and by people from other European countries. After that, question B37 was asked as shown below.

Questions on immigration (Open in new window: B37 - B38 - B39 - B40)

Based on these questions, access the program Survey Quality Predictor (SQP) and add the missing spaces about the quality predictions of these questions in the table below. For this particular illustration, we have chosen the Netherlands case for reasons that will become clear later on.3 The table provides the information necessary to find the questions in SQP, which are already coded and authorized.

Input for exercise*
Question r2 v2 q2 m**
B31 (Allow) 0.840 0.763
B38 (Economy) 0.875 0.354
B39 (Culture) 0.777 0.641
B40 (Better) 0.878 0.349

*SQP Study = ESS Round 3, SQP Country = Netherlands, SQP Language = Dutch
**m is defined as mi = √(1-vj2).


In the following, a description of the steps necessary to obtain the results in SQP is presented:

  1. Access, and register if you are not already a user.
  2. On the Home page of SQP, follow the link: ‘View all questions that are currently available >>’. This link will take you to the list of all available questions in SQP.
  3. On the left side of the screen, users can easily search for the desired questions by filtering the known details (Study, Country and Language). Furthermore, in the ‘Containing text’ option, you can write the name of the question of interest (e.g. B37).
  4. Introduce the details. For question B37, select from the browser list ‘All Studies’ the study ‘ESS Round 3’; from the browser list ‘All Languages’, select the language ‘Dutch; from the browser list ‘All Countries’, select the country ‘Netherlands’, and, in the ‘Containing text’ option, enter the question name ‘B37’.
  5. The question list is reduced to just one question with the specified characteristics. Select the question and you will be able to see the question text and the authorized prediction (i.e. the prediction made by trained coders under the supervision of the SQP team). Click ‘View Prediction detail’ to obtain the quality values of reliability, validity and quality.
  6. From the SQP quality predictions screen, you can get the necessary information to fill in the table. Specifically, for this question B37, we get 0.840, 0.908 and 0.763 for the reliability, validity and quality values, respectively.
  7. Repeat the exact same procedure (steps 3 to 6) to obtain the coefficients for the other questions. The only information that should be changed each time is the question name in the ‘Containing text’ option.
  8. The method effect (m) coefficient can be computed as: mi = √(1-v2).
    mB37 = √(1-0.908) = 0.303
    mB38 = √(1-0.875) = 0.354
    mB39 = √(1-0.825) = 0.418
    mB40 = √(1-0.878) = 0.349
Question r2 v2 q2 M
B31 (Allow) 0.840 0.908 0.763 0.303
B38 (Economy) 0.802 0.875 0.702 0.354
B39 (Culture) 0.777 0.825 0.641 0.418
B40 (Better) 0.728 0.878 0.639 0.349

*SQP Study = ESS Round 3, SQP Country = Netherlands, SQP Language = Dutch

Go to next chapter >>

Page 4