9.3 The likelihood ratio, Wald and Lagrange multiplier tests

Maximum likelihood supplies three general tests of a hypothesis \(H_0\colon c(\theta ) = 0\) with \(q\) restrictions. Let \(\hat \theta \) be the unrestricted MLE and \(\tilde \theta \) the MLE under \(H_0\).

The likelihood ratio test compares the two maximised log-likelihoods: \(\text {LR} = 2\bigl [\ell (\hat \theta ) - \ell (\tilde \theta )\bigr ]\). It needs both estimates.
The Wald test asks whether the unrestricted estimate nearly satisfies the restrictions: it measures \(c(\hat \theta )\) against its estimated variance, \(W = c(\hat \theta )'\bigl [\widehat \Var \, c(\hat \theta )\bigr ]^{-1}c(\hat \theta )\). It needs only the unrestricted estimate.
The Lagrange multiplier, or score, test asks whether the restricted estimate is nearly a maximum of the unrestricted likelihood: it measures the score at the restricted estimate, \(\text {LM} = s(\tilde \theta )'I(\tilde \theta )^{-1}s(\tilde \theta )\). It needs only the restricted estimate.

Under \(H_0\) all three are asymptotically \(\chi ^2_q\), and they are asymptotically equivalent. The choice between them is usually practical. The LM test is attractive when the restricted model is simple and the unrestricted one is not, which is the case for most diagnostic tests (heteroscedasticity, autocorrelation, omitted variables), since the null model is the plain regression already estimated.

Theorem 9.3.1. For \(q\) linear restrictions \(R\beta = r\) in the normal linear model, with restricted and unrestricted residual sums of squares \(\text {RSS}_R\) and \(\text {RSS}_U\), \[ W = \frac {n(\text {RSS}_R - \text {RSS}_U)}{\text {RSS}_U}, \qquad \text {LR} = n\log \frac {\text {RSS}_R}{\text {RSS}_U}, \qquad \text {LM} = \frac {n(\text {RSS}_R - \text {RSS}_U)}{\text {RSS}_R} , \] and in every sample \[ W \geq \text {LR} \geq \text {LM} . \] Also \(W = \frac {n}{n-k}qF\), where \(F\) is the statistic of Theorem 7.4.1, and \(\text {LM} = nR^2\) from the regression of the restricted residuals on all the regressors \(X\), when the restricted model contains an intercept.

Proof. LR: by Theorem 9.2.1(iii), \(\ell (\hat \theta ) - \ell (\tilde \theta ) = \frac n2\log (\text {RSS}_R/\text {RSS}_U)\), since the constants cancel.

\(W\): the Wald statistic for \(R\beta = r\) uses \(\widehat \Var (Rb) = \hat \sigma ^2R(X'X)^{-1}R'\) with the unrestricted \(\hat \sigma ^2 = \text {RSS}_U/n\). By Theorem 7.4.1, the quadratic form \((Rb - r)'[R(X'X)^{-1}R']^{-1}(Rb - r)\) equals \(\text {RSS}_R - \text {RSS}_U\), so \(W = (\text {RSS}_R - \text {RSS}_U)/(\text {RSS}_U/n)\). Comparing with \(F = \bigl ((\text {RSS}_R - \text {RSS}_U)/q\bigr )/\bigl (\text {RSS}_U/(n - k)\bigr )\) gives \(W = \frac {n}{n-k}qF\).

LM: the LM statistic in this model is the same quadratic form evaluated with the restricted variance estimate \(\tilde \sigma ^2 = \text {RSS}_R/n\), giving \(\text {LM} = (\text {RSS}_R - \text {RSS}_U)/(\text {RSS}_R/n)\). For the \(nR^2\) form: let \(e_*\) be the restricted residuals. Since \(Y - e_* = Xb_*\) lies in the column space of \(X\), \(Me_* = MY = e\). So the regression of \(e_*\) on \(X\) has residual sum of squares \(e'e = \text {RSS}_U\). If the restricted model has an intercept, \(e_*\) has mean zero, its total sum of squares is \(e_*'e_* = \text {RSS}_R\), and \(R^2 = 1 - \text {RSS}_U/\text {RSS}_R\), so \(nR^2 = \text {LM}\).

Inequality: write \(x = (\text {RSS}_R - \text {RSS}_U)/\text {RSS}_U \geq 0\). Then \(W = nx\), \(\text {LR} = n\log (1 + x)\) and \(\text {LM} = nx/(1 + x)\). For \(x \geq 0\), \(x \geq \log (1 + x) \geq x/(1 + x)\): the first because \(\log (1 + x)\) is concave with slope \(1\) at \(0\), the second because \(\log (1 + x) = \int _0^x\frac {dt}{1 + t} \geq \frac {x}{1 + x}\). □

The inequality is a finite-sample fact with a practical consequence. Since all three have the same asymptotic \(\chi ^2_q\) distribution, a Wald test will reject more often than an LR test, and an LR test more often than an LM test. With a borderline result the conclusion can depend on which test is used, and the honest course is to report which one it was.

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.