9.1 The likelihood principle
Let the data \(y\) have density \(f(y; \theta )\), depending on a parameter vector \(\theta \) of dimension \(p\). Regarded as a function of \(\theta \) for the observed \(y\), \(L(\theta ) = f(y; \theta )\) is the likelihood, and \(\ell (\theta ) = \log L(\theta )\) the log-likelihood. The maximum likelihood estimator (MLE) \(\hat \theta \) maximises \(\ell \). Define \[ s(\theta ) = \frac {\partial \ell }{\partial \theta } \quad \text {(the score)}, \qquad I(\theta ) = -E\left [\frac {\partial ^2\ell }{\partial \theta \,\partial \theta '}\right ] \quad \text {(the information matrix)} . \] Under “regularity conditions”, chiefly that the support of \(f\) does not depend on \(\theta \) and that differentiation and integration can be interchanged, these quantities have two key properties.
Theorem 9.1.1. At the true parameter value, \(E[s(\theta )] = 0\) and \(\Var [s(\theta )] = E[s(\theta )s(\theta )'] = I(\theta )\). The second statement is the information matrix equality.
Proof. Differentiate \(\int f(y; \theta )\,dy = 1\) with respect to \(\theta \): \[ 0 = \int \frac {\partial f}{\partial \theta }\,dy = \int \frac {\partial \log f}{\partial \theta }f\,dy = E(s) . \] Differentiate \(\int s(\theta )f\,dy = 0\) once more with respect to \(\theta '\): \[ 0 = \int \frac {\partial s}{\partial \theta '}f\,dy + \int s\frac {\partial f}{\partial \theta '}\,dy = E\left [\frac {\partial ^2\ell }{\partial \theta \,\partial \theta '}\right ] + E(ss') , \] since \(\partial f/\partial \theta ' = f\,s'\). Hence \(E(ss') = I(\theta )\), and since \(E(s) = 0\) this is \(\Var (s)\). □
Theorem 9.1.2 (Cramér–Rao lower bound). Under the same conditions, if \(\tilde \theta \) is any unbiased estimator of \(\theta \), then \(\Var (\tilde \theta ) - I(\theta )^{-1}\) is positive semi-definite.
Proof. Unbiasedness says \(\int \tilde \theta (y)f(y; \theta )\,dy = \theta \) for every \(\theta \). Differentiating with respect to \(\theta '\), \(\int \tilde \theta \,s'f\,dy = I_p\), that is \(E(\tilde \theta s') = I_p\). Since \(E(s) = 0\), this is \(\Cov (\tilde \theta , s) = I_p\). The joint covariance matrix \[ \Var \begin {pmatrix} \tilde \theta \\ s \end {pmatrix} = \begin {pmatrix} \Var (\tilde \theta ) & I_p \\ I_p & I(\theta ) \end {pmatrix} \] is psd. Premultiply by \((I_p\ \ -I(\theta )^{-1})\) and postmultiply by its transpose: the result, \[ \Var (\tilde \theta ) - 2I(\theta )^{-1} + I(\theta )^{-1}I(\theta )I(\theta )^{-1} = \Var (\tilde \theta ) - I(\theta )^{-1}, \] is psd, being of the form \(A'\Sigma A\) with \(\Sigma \) psd. □
No unbiased estimator can have smaller variance than \(I(\theta )^{-1}\). An unbiased estimator that attains the bound is efficient in a far stronger sense than Gauss–Markov provides: it beats all unbiased estimators, linear or not. Usually no estimator attains the bound exactly in finite samples, but maximum likelihood attains it in the limit.
Theorem 9.1.3 (Asymptotic properties of the MLE). Let \(y_1, \dots , y_n\) be independent and identically distributed with density \(f(y; \theta )\), let \(I_1(\theta )\) be the information in one observation, so that \(I(\theta ) = nI_1(\theta )\), and assume regularity conditions. Then the MLE is consistent, and \[ \sqrt n(\hat \theta - \theta ) \xrightarrow {d} N\bigl (0, I_1(\theta )^{-1}\bigr ) . \] Moreover, for any differentiable function \(g\), the MLE of \(g(\theta )\) is \(g(\hat \theta )\) (invariance).
Sketch of proof. Consistency rests on the fact that \(\frac 1n\ell (\theta )\) converges to \(E[\log f(y; \theta )]\), which is maximised at the true \(\theta \) (by Jensen’s inequality, which gives the information inequality), and on conditions that make the maximiser of the limit the limit of the maximisers. For the distribution, expand the score about the true \(\theta _0\): by the mean value theorem, \(0 = s(\hat \theta ) = s(\theta _0) + H(\bar \theta )(\hat \theta - \theta _0)\), with \(H\) the matrix of second derivatives and \(\bar \theta \) between \(\hat \theta \) and \(\theta _0\). So \[ \sqrt n(\hat \theta - \theta _0) = \left [-\tfrac 1nH(\bar \theta )\right ]^{-1}\tfrac {1}{\sqrt n}s(\theta _0) . \] \(s(\theta _0)\) is a sum of \(n\) independent terms with mean zero and variance \(I_1\) (Theorem 9.1.1), so \(n^{-1/2}s(\theta _0) \xrightarrow {d} N(0, I_1)\) by the CLT. By the law of large numbers and consistency, \(-\frac 1nH(\bar \theta ) \xrightarrow {p} I_1\). Slutsky’s theorem gives the limit \(N(0, I_1^{-1}I_1I_1^{-1}) = N(0, I_1^{-1})\). Invariance holds because maximising over \(\theta \) and over a one-to-one reparametrisation give corresponding points; for a general \(g\) it is taken as the definition. □
So the MLE is consistent, asymptotically normal, and asymptotically efficient: its large-sample variance \(I(\theta )^{-1}\) is the Cramér–Rao bound. The regularity conditions matter. They fail, for instance, when a parameter determines the range of the data, as for the uniform distribution on \((0, \theta )\), and the theory then breaks down. In regression, the observations are independent but not identically distributed, and the same conclusions hold under conditions on the regressors.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.