2.6 Maximum likelihood estimation

Least squares needs only A1–A3. If we add the normality assumption A4, we can estimate the parameters by the other standard method, maximum likelihood. Reassuringly, it gives the same line.

Theorem 2.6.1. Under A1–A4, the maximum-likelihood estimators of \(\beta _0\) and \(\beta _1\) are the least-squares estimators \(\hat \beta _0\) and \(\hat \beta _1\), and the maximum-likelihood estimator of \(\sigma ^2\) is \[ \tilde \sigma ^2 = \frac {SS_{\mathrm {Res}}}{n} , \] which is biased: \(E(\tilde \sigma ^2) = \dfrac {n-2}{n}\,\sigma ^2\).

Proof. Under A1–A4 the \(y_i\) are independent with \(y_i \sim N(\beta _0 + \beta _1 x_i, \sigma ^2)\), so the likelihood is the product of normal densities, \begin {align*} L(\beta _0, \beta _1, \sigma ^2) &= \prod _{i=1}^n \frac {1}{\sqrt {2\pi \sigma ^2}} \exp \left \{ -\frac {(y_i - \beta _0 - \beta _1 x_i)^2}{2\sigma ^2} \right \} \\ &= (2\pi \sigma ^2)^{-n/2} \exp \left \{ -\frac {S(\beta _0, \beta _1)}{2\sigma ^2} \right \}, \end {align*}

where \(S\) is the least-squares criterion. The log-likelihood is \[ \ell = -\frac n2 \log (2\pi ) - \frac n2 \log \sigma ^2 - \frac {S(\beta _0, \beta _1)}{2\sigma ^2} . \] For any fixed \(\sigma ^2\), \(\ell \) is largest when \(S(\beta _0, \beta _1)\) is smallest, that is, at \(\hat \beta _0, \hat \beta _1\), whatever the value of \(\sigma ^2\). Putting these in, \(S\) becomes \(SS_{\mathrm {Res}}\), and \[ \frac {\partial \ell }{\partial \sigma ^2} = -\frac {n}{2\sigma ^2} + \frac {SS_{\mathrm {Res}}}{2\sigma ^4} = 0 \quad \text {gives}\quad \tilde \sigma ^2 = \frac {SS_{\mathrm {Res}}}{n} . \] (The derivative is positive below this value and negative above it, so it is a maximum.) Finally, by Theorem 2.5.3, \(E(\tilde \sigma ^2) = E(SS_{\mathrm {Res}})/n = (n-2)\sigma ^2/n\). □

The bias is small when \(n\) is large, but for small samples it matters: with the five advertising observations, \(\tilde \sigma ^2 = 1.10/5 = 0.22\) against \(\hat \sigma ^2 = 0.3667\). We always use the unbiased \(\hat \sigma ^2 = MS_{\mathrm {Res}}\).

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.