Survey Quality Predictor: SQP

Based on all this knowledge, this algorithm has been used to develop the computer program Survey Quality Predictor SQP 2.0,1 which generates predictions of the quality of survey questions [Obe11].

The Survey Quality Predictor 2.0 is a tool used to obtain predictions about the quality of survey questions. Nowadays, it provides a large database of more than 65,000 survey questions from many different questionnaires, in many different languages and about many different topics. The quality prediction is obtained by coding the characteristics in SQP. Currently, quality predictions are available for more than 10,000 survey questions. Besides, the Survey Quality Predictor is an open source tool under constant development, as the database is created with the collaboration of users. SQP users can be participants in the development of the program by adding new questions, coding their characteristics and obtaining quality predictions, or by coding the characteristics of any question already available in the database and obtaining quality predictions. It is important to note that, not only are the survey questions introduced by all users available to all SQP users, the predictions obtained by any user are also publicly available.

Thus, the Survey Quality Predictor has become a useful tool for questionnaire designers and researchers for gathering quality information about survey questions. This information is obtained without collecting new data. The only thing the researcher has to do is to code the characteristics of the questions. This is a major advantage compared with the procedures so far available for obtaining estimates of the quality of questions. In order to learn the standard process of the coding, we suggest taking a look at SQP 2.0.

The SQP team has made the ESS questionnaires available in the SQP database and organized the coding of the characteristics of the questions in the MTMM experiments in all the countries and languages participating since ESS Round 1. Thus, besides having more than 62,000 ESS questions from more than 20 different languages, an authorized quality prediction is available for more than 8,500 ESS questions. Since, on the other hand, other SQP users have coded questions as well, it is important to differentiate between user codes and authorized codes. The authorized codes can be trusted because they have been coded in different languages by native coders under the training and supervision of SQP members. Thus, if the question of interest is coded but not authorized, we suggest checking the coding before using the predictions.

To conclude, this means that researchers can now obtain, via SQP, a prediction of the quality of most ESS questions, but also of other new questions, without incurring costs other than the time required to introduce the question in the program and to code it. So, the program SQP makes it possible to obtain quality predictions for nearly all questions. A major problem for the researchers has thereby been solved, i.e. one only needs the estimates of the quality of all variables in the study in order to be able to correct for measurement errors in the analysis.

We say ‘nearly all questions’ because it is difficult to design MTMM experiments for background variables that also contain measurement errors, as has been shown by [Sch14] and [Alw07]. However, quality information about background variables can be obtained from [Alw07], who used quasi-simplex models. Combining the SQP predictions and the estimates of [Alw07] with respect to background variables, it is possible to obtain estimates of the quality of all questions used in survey research and it is therefore also possible to correct the correlations between all variables.

Furthermore, we should realize that SQP is not able to predict the quality of questions for countries in which no MTMM experiments have been done so far. At the moment, SQP can provide predictions for the following countries: Austria, Belgium, Czech Republic, Denmark, Estonia, Finland, France, Germany, Great Britain, Ireland, Netherlands, Norway, Poland, Portugal, Slovakia, Slovenia, Spain, Sweden, Switzerland, Ukraine and the United States.