12.2 Least squares and the normal equations

The least-squares estimate \(\hat {\boldsymbol \theta }\) minimises \[ S(\boldsymbol \theta ) = \sum _{i=1}^n\bigl [y_i - f(\mathbf {x}_i, \boldsymbol \theta )\bigr ]^2 = \bigl \|\mathbf {y} - \mathbf {f}(\boldsymbol \theta )\bigr \|^2 , \] where \(\mathbf {f}(\boldsymbol \theta )\) is the vector of the \(f(\mathbf {x}_i, \boldsymbol \theta )\). Let \(\mathbf {F}(\boldsymbol \theta )\) be the \(n \times p\) matrix of derivatives, \(F_{ij} = \partial f(\mathbf {x}_i, \boldsymbol \theta )/\partial \theta _j\). In a linear model \(\mathbf {f} = \mathbf {X}\boldsymbol \theta \) and \(\mathbf {F} = \mathbf {X}\); in general \(\mathbf {F}\) plays the part of the model matrix, but it changes with \(\boldsymbol \theta \).

Theorem 12.2.1.

(i): The least-squares estimate satisfies the normal equations \[ \mathbf {F}(\hat {\boldsymbol \theta })^\top \bigl [\mathbf {y} - \mathbf {f}(\hat {\boldsymbol \theta })\bigr ] = \mathbf {0} : \] the residual vector is orthogonal to the columns of \(\mathbf {F}(\hat {\boldsymbol \theta })\).
(ii): If the errors are normal, the least-squares estimate is the maximum-likelihood estimate of \(\boldsymbol \theta \), and the maximum-likelihood estimate of \(\sigma ^2\) is \(S(\hat {\boldsymbol \theta })/n\).

Proof. (i) \(\partial S/\partial \theta _j = -2\sum _i[y_i - f(\mathbf {x}_i, \boldsymbol \theta )]F_{ij}\), so the gradient is \(-2\mathbf {F}^\top (\mathbf {y} - \mathbf {f})\), which is zero at a minimum.

(ii) The log-likelihood is \(-\frac n2\log (2\pi \sigma ^2) - S(\boldsymbol \theta )/(2\sigma ^2)\), exactly as in the proof of Theorem 2.6.1. For any \(\sigma ^2\) it is largest where \(S\) is smallest, and maximising over \(\sigma ^2\) then gives \(S(\hat {\boldsymbol \theta })/n\). □

Unlike the linear case, the normal equations are non-linear in \(\hat {\boldsymbol \theta }\), may have several solutions (local minima, maxima and saddle points of \(S\)), and must be solved numerically.

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.