An illustration of Maximum Likelihood (ML) estimation

The example is inspired by a similar example in [Won77].

Let us assume that we have been assigned the task of estimating the quality of a production line, in other words to estimate P(Defect), the probability that a randomly chosen product is defective. Assume further that we have drawn a sample of five products and found three to be defective. A common sense estimate of P is 3/5 or 0.6. Let us try to estimate P by using a maximum likelihood principle. First, we need the likelihood function that is necessary to evaluate the estimates. The number of defect products can be seen as resulting from a binomial experiment with n=5 trials. The probability distribution of the random variable X (number of defective products) is the binomial distribution:

We can manually try out various values of P, choosing the one that returns the maximum value of the likelihood function. In statistical software, this is done by an iterated algorithm. Let us start with P=0.10. What is the probability of observing three defective products out of five, given P=0.10?

L(p̂) = 10*0.13*0.92 = 0.008

Table 2.2 shows the results of the likelihood function for probabilities (P) varying from 0 to 1. The function peaks at P=0.6 as shown in Figure 2.2. Thus, the use of the maximum likelihood algorithm yields the same result as our common sense estimate. For more complex situations, there are no equivalents to the common sense estimate and the likelihood functions are much more complex than in our example.

Table 2.2. The likelihood of observing three defective products out of five for various values of P
P 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
L(P) 0 0.008 0.051 0.132 0.23 0.312 0.346 0.309 0.205 0.073 0

Figure 2.2. An illustration of maximum likelihood estimation

Estimating OLS models in statistical procedures for multilevel models


In SPSS can linear multilevel models be estimated using the Linear Mixed Models procedure. For most purposes, writing the syntax is preferable to using the menus. Let us estimate the final model without age squared and the interaction term with the minimum of required text:

MIXED wage BY egp WITH edyears age female
/FIXED= edyears age female egp | SSTYPE(3)

The structure of the command is different from REGRESSION. The dependent variable follows the MIXED keyword, and factors (categorical covariates) follow the BY keyword. This means that we do not have to create a set of dummy variables for classes manually, as long as we are satisfied with having the last class as the reference category. The continuous covariates (or dummy variables) follow the WITH keyword. The interaction term is included as in REGRESSION, but we do not need to compute the variable manually. Instead, we could have replaced ‘edfem’ with ‘edyears*female’. The FIXED subcommand defines the (fixed) regression coefficients and their order in the output table. The estimation method is restricted maximum likelihood (REML) and the last command prints the solution. The most interesting tables are included below. The coefficients and the t-test are identical to those from REGRESSION. There are a few differences. Because of the difference in the estimation procedures, standardized coefficients are not available in MIXED and nor is the R square . The small table below shows estimates of the total residual variance. In the multilevel models to come, this table will contain more interesting information.

Table 2.3. Estimates of Fixed Effectsa - SPSS output

Table 2.4. Estimates of Covariance Parametersa - SPSS output


In Stata, two procedures - XTREG and XTM IXED - can be used to estimate multilevel regression. In models of cross-sectional data, the XTMIXED procedure will be used throughout this module, since it is the most general one and allows the estimation of all ensuing models. The syntax for XTMIXED and the output is shown below. Note the ‘variance’ at the end that ensures that the variance components or random-effects parameters are reported in the variance scale rather than the default standard deviations. The estimates are identical to the ones from SPSS.

Table 2.5. XTMIXED estimates - Stata output

Go to next chapter >>