Sample sizes in multilevel analysis

This section on considerations relating to sample sizes in multilevel analysis builds on Hox (2010, Chapter 12). It translates into two questions: how large samples are necessary for multilevel analysis and what sample size is required to obtain a certain level of statistical power. We will focus on the former.

ML methods of estimation are asymptotic, which means that the sample size should be large. It is not clear, however, how the requirement for a large sample should be interpreted, nor whether it applies equally to all levels? The accuracy of estimates with varying sample sizes can be studied by simulation methods, and Hox summarizes current knowledge about this topic.

The regression coefficients are largely unbiased irrespective of which of the estimation methods mentioned is used. The standard errors of the regression coefficients are severely biased by using OLS estimation, and the ML estimates are slightly downwardly biased if the number of groups (level 2 units) is less than 50 and the normality assumption is violated.

For the variance components, the situation varies. The variance in level 1 residual errors is normally very accurately estimated. The same goes for level 2 variance in analyses with 100 or more groups (level 2 units). Simulations indicate that the level 2 variances will be satisfactory estimated with 30 groups and underestimated if the number of groups is as low as 10. This means that multilevel analysis with countries as level 2 units will most likely yield downwardly biased estimates of country level variation.

Since multilevel analysis involves two or more levels, questions concerning optimal sample sizes are difficult to answer, and the best advice will also depend on the purpose. Hox mentions Kreft’s 30/30 rule, which means 30 groups with a least 30 individuals in each. This could be sufficient for the estimation of the regression coefficients but inadequate for other purposes. If it is cross-level interactions that are of interest, Hox recommends the 50/20 rule: 50 groups with 20 or more in each group. If there is strong interest in the random part, the advice is 100 groups with a minimum of ten in each.

Go to next page >>