Chapter 5: Taking the Sample Design into Account: Design Effects

The need for adequate consideration of the effects of a sample design on the precision of estimators is being recognised in an increasing number of sample survey projects. These projects make use of a concept called design effect that can serve as a measure of variance inflation in the estimator, due to a departure from simple random sampling. The European Social Survey was the first general social survey to make explicit use of design effects already at the planning stage [Ess05a]. In the ESS, each participating country is responsible for its sample meeting certain pre-defined quality criteria. One of these criteria concerns the precision of estimators: the samples of all participating countries shall yield estimators of comparable precision1. Design effects play a crucial role in the planning of samples that will yield estimators with these properties.

The foundations for sampling in Europe-wide surveys like the ESS are quite diverse. In some countries, such as Sweden, Norway or Finland, researchers are allowed to draw a sample directly from population registers. In other countries, such as Portugal, Spain or Poland, access to population registers is either limited or not possible at all. This diversity of sampling frames results in a diversity of sample designs. Whereas in countries in the first group, a simple random sample or a stratified random sample of contact persons can be drawn directly, this is not possible in the second group of countries due to the structure of the sampling frame. These countries often have to resort to more complex sample designs such as cluster or multi-stage sample designs. It is an empirical fact, however, that persons socialised within the same social context (e.g. living in the same neighbourhood or municipality) are more similar to each other than to persons who are socialised in a different social context in their responses to many questions in a general social survey like the ESS. This homogeneity can have a negative effect on the precision of estimators.

The accuracy of an estimator calculated using data from a simple random sample differs from the quality of the same estimator calculated on the basis of a cluster sample design described above, given that the two samples are of the same size. Nevertheless, all samples in the ESS have to comply with the aforementioned quality standards in terms of the precision of estimators. The question, then, is how to design different samples so that these criteria are met under the practical restriction of divergent sampling frames.