A simple procedure to correct for measurement errors in survey research

By Anna DeCastellarnau and Willem Saris

Introduction

Although most applied researchers believe that survey data contain measurement errors, very few correct these errors. In principle, the reason for this omission cannot be that the procedures are not known because they were already developed in the 20th century [Gol73]. One of the reasons for this omission is probably that these procedures make it necessary to collect multiple indicators for all variables in the study in order to correct for measurement errors. This doubles or triples the response burden for the respondents, increases the costs of the research and makes the analysis rather complex. The purpose of this text is therefore to illustrate how corrections can be made in a very simple way and to show that researchers can and should always correct for these errors.

The procedure for correcting for measurement errors can be very simple if we have estimates of the size of these errors in the data or the quality of the measures (i.e. questions) used. In that case, the correlation or covariance matrices can be corrected for measurement errors, after which, using these matrices as data input, the analysis is the same as the analysis without correction for measurement error. However, this approach is only possible if estimates of the quality of all the measures in the study are available. This has been a problem so far. With the development of the program Survey Quality Predictor (SQP), however, predictions can be obtained of the quality of all questions in more than 20 languages [Sar14]. Due to the development of SQP, a very simple procedure for correction for measurement error can be suggested.

In order to illustrate this procedure, we will first show that survey analyses without correction for measurement errors cannot be trusted. In doing so, we will also illustrate how large differences there can be between the results of analyses with and without correction for measurement errors. After that, we will explain how the quality of questions can be obtained. Finally, we will show how this information can be used in the analysis to correct for measurement errors.

In this module, as an illustration of correction for measurement errors, we will use as an example one of the topics introduced in one of the rotating modules of Round 6 of the European Social Survey (ESS): Europeans’ understandings and evaluations of democracy [Kri10].

In their proposal, the researchers proposed to study the factors that explain varying levels of satisfaction with democracy throughout Europe. In order to keep it simple, we have only selected some aspects mentioned in the module proposal. The dependent variable has been present in all rounds of the ESS. It measures the concept ‘satisfaction with democracy’ (B23) and has been measured as follows in the British version of ESS Round 6:1

Table 0.1: The formulation of the question ‘satisfaction with democracy’ in the British questionnaire of ESS Round 6.

The explanatory variables in the proposal consisted of a large series of questions measuring the evaluation of specific, important aspects of democracy in the country of interest. For the purpose of this module, we have chosen only three aspects: ‘free and fair elections’ (E17), ‘freedom to criticize’ (E20) and ‘equality by law’ (E25), which are measured as indicated in Table 0.2 in the British version of the ESS Round 6:

Table 0.2: The formulation of the questions for three variables measuring ‘evaluation of democracy’ in the British questionnaire of ESS Round 6.

The question about satisfaction with democracy was asked at the beginning of the questionnaire (Module B), while questions about the explanatory variables were asked towards the end (Module E).

Furthermore, we have chosen as control variables two variables that are often used in this context. They are ‘left-right placement’ (B19) and ‘household income’ (F41).

Table 0.3: The formulation of the question for the variable ‘left-right placement’ in the British questionnaire of ESS Round 6.

Table 0.4a: The formulation of the question for the variable ‘household income’ in the British questionnaire of ESS Round 6.

Table 0.4b: The formulation of the showcard for the variable ‘household income’ in the British questionnaire of ESS Round 6

Having presented the variables we will use in this module, the following structure will be used to present our arguments in favour of correcting for measurement errors. In the first chapter, we will illustrate why researchers should not trust results without correction for measurement errors, showing how large the errors can be by comparing the effects of different formulations of the questions for the same concepts. In the second chapter, we will discuss where these errors come from and what the relationship is between these errors and the quality of survey questions. In the third chapter, we will explain how the quality estimates can be obtained using different procedures, and we will also show how the program Survey Quality Predictor (SQP) can be applied to obtain the information required to correct for measurement errors. In the fourth chapter, the quality estimates will be used to correct the correlation matrix for measurement errors. Finally, in the last three chapters, we will demonstrate how corrections for measurement errors can be made in a simple way by performing estimations, with and without correction for measurement errors, of regression and causal models based on the correlation matrix (Chapter 5), of a causal model on the basis of a covariance matrix to obtain unstandardized coefficients (Chapter 6), and of causal models with complex concepts (Chapter 7). All the analyses are presented using both LISREL and Stata.

Contents of the module

Chapter 1 Why should we correct for measurement errors?

The goal of this chapter is to demonstrate why the results researchers usually obtain from survey research without correction for measurement errors cannot be trusted. This is done by showing the effects of different formulations of questions measuring the same concepts on the response distributions and the correlations between variables.

Chapter 2 The relationship between errors and the quality of questions

In this chapter, by developing a response model, we indicate the different types of measurement errors that can be expected in survey research. This model also allows us to indicate the link between the size of these errors and the quality of survey questions. This is relevant because these quality estimates will be used to correct for measurement errors.

Chapter 3 The estimation of the quality of survey questions

So far, the quality of survey questions has been evaluated through elaborate studies and experiments using multiple indicators for the relevant variables. In this chapter, we present an alternative to these experiments that allows researchers to obtain a prediction of the quality coefficients required to correct for measurement errors without collecting new data. This alternative is the Survey Quality Predictor (SQP) program.

Chapter 4 Correction of correlations for measurement errors

Now that we have seen that we can obtain estimates of the quality of survey questions, we can show what the effect of the lack of quality is, i.e. the errors in the correlations between variables, but we will also illustrate how we can correct the correlations with respect to these measurement errors.

Chapter 5 Estimation of regression and causal models analysis with and without correction for measurement errors

In this chapter, we are going to illustrate, for regression and causal models, how correcting for measurement errors has an impact on the results obtained. This will be done by presenting the results of both analyses with and without correction for measurement errors. It will be shown that the procedures for both analyses are exactly the same, except for the correlation matrix used as the basis for the analysis. The chapter will also demonstrate how large the differences are between the analyses with or without correction for measurement errors.

Chapter 6 Estimation of the unstandardized effects in causal models

So far, we have applied the quality estimates to correct the correlation matrix in order to obtain the standardized coefficients. In this chapter, we want to demonstrate that analyses of causal models with correction for measurement errors can also be simply performed on covariance matrices to obtain the unstandardized coefficients.

Chapter 7 Correction for measurement error in causal models with complex concepts

In this chapter, we will illustrate that correction for measurement errors can be done not just for concepts measured by single questions but also for complex concepts measured by composite scores based on several questions. We will compute the quality of a composite score based on the quality of the variables that formed the basis for the composite score. After correcting the correlation matrix, the analysis can be performed as illustrated in the previous chapters, in the same way with and without correction for measurement errors.

Appendix 1: Coding in SQP

In this appendix, we use the questions about the level of satisfaction with democracy in Great Britain to go through the SQP coding process for each of the characteristics of these survey questions. It would be a good exercise to replicate the steps presented here for the same survey questions in other languages to learn and understand the coding process of SQP.

Appendix 2: R inputs

Users can already go through the module using either SPSS or Stata. However, for those of you who are more familiar with R we also provide you, in this appendix with two R syntaxes; one for the regression model (Chapter 5) and another for the causal model (Chapter 6). These can easily be used and adapted to other models by correcting the syntax.

Go to first chapter >>

Footnotes

References