6.2 Least squares
Theorem 6.2.1. Under (A4), the residual sum of squares \(S(b) = (Y - Xb)'(Y - Xb)\) has a unique minimum at \[ b = (X'X)^{-1}X'Y , \] the solution of the normal equations \(X'Xb = X'Y\).
Proof. Expand \(S(b) = Y'Y - 2b'X'Y + b'X'Xb\), using \(Y'Xb = b'X'Y\), which is a scalar. By Lemma 5.1.8, \[ \frac {\partial S}{\partial b} = -2X'Y + 2X'Xb , \] which is zero exactly when \(X'Xb = X'Y\). The matrix \(X'X\) is psd, since \(c'X'Xc = (Xc)'(Xc) \geq 0\), and \(Xc = 0\) only for \(c = 0\) because the columns of \(X\) are linearly independent. So \(X'X\) is pd and therefore invertible, giving the unique solution \(b = (X'X)^{-1}X'Y\). The Hessian \(2X'X\) is pd, so \(S\) is strictly convex and the stationary point is the global minimum. A direct argument: for any \(\tilde b\), \begin {align*} S(\tilde b) &= \bigl (e + X(b - \tilde b)\bigr )'\bigl (e + X(b - \tilde b)\bigr )\\ &= S(b) + (b - \tilde b)'X'X(b - \tilde b) \geq S(b) , \end {align*}
where \(e = Y - Xb\) and the cross terms vanish because \(X'e = X'Y - X'Xb = 0\). □
With \(P = X(X'X)^{-1}X'\) and \(M = I - P\) as in Example 5.1.7, the fitted values and residuals are \[ \hat Y = Xb = PY, \qquad e = Y - \hat Y = MY = MU . \]
Theorem 6.2.2. The least squares residuals satisfy \(X'e = 0\): they are orthogonal to every regressor. Consequently \(\hat Y'e = 0\), and if the model contains an intercept, \(\iota 'e = \sum e_i = 0\), the fitted plane passes through the point of means, and \(\bar {\hat Y} = \bar Y\).
Proof. \(X'e = X'MY = (MX)'Y = 0\). Then \(\hat Y'e = b'X'e = 0\). If \(\iota \) is the first column of \(X\), the first row of \(X'e = 0\) is \(\iota 'e = 0\). Dividing \(\iota 'Y = \iota 'Xb + \iota 'e = \iota 'Xb\) by \(n\) gives \(\bar Y = b_1 + b_2\bar X_2 + \dots + b_k\bar X_k\). □
Example 6.2.3. Five observations on \(Y\) and two regressors:
| \(X_2\) | 1 | 2 | 3 | 4 | 5 |
| \(X_3\) | 1 | 3 | 2 | 5 | 4 |
| \(Y\) | 3 | 6 | 7 | 11 | 13 |
The means are \(\bar X_2 = \bar X_3 = 3\) and \(\bar Y = 8\). In deviations, \(x_2 = (-2, -1, 0, 1, 2)\), \(x_3 = (-2, 0, -1, 2, 1)\) and \(y = (-5, -2, -1, 3, 5)\), and \begin {gather*} \sum x_2^2 = 10, \quad \sum x_3^2 = 10, \quad \sum x_2x_3 = 8,\\ \sum x_2y = 25, \quad \sum x_3y = 22, \quad \sum y^2 = 64 . \end {gather*} The slope coefficients solve the normal equations in deviation form (Theorem 6.4.2 below): \[ \begin {pmatrix} 10 & 8 \\ 8 & 10 \end {pmatrix}\begin {pmatrix} b_2 \\ b_3 \end {pmatrix} = \begin {pmatrix} 25 \\ 22 \end {pmatrix} \quad \Longrightarrow \quad \begin {pmatrix} b_2 \\ b_3 \end {pmatrix} = \frac {1}{36}\begin {pmatrix} 10 & -8 \\ -8 & 10 \end {pmatrix}\begin {pmatrix} 25 \\ 22 \end {pmatrix} = \begin {pmatrix} 74/36 \\ 20/36 \end {pmatrix} . \] So \(b_2 = 2.0556\), \(b_3 = 0.5556\) and \(b_1 = \bar Y - b_2\bar X_2 - b_3\bar X_3 = 8 - 3(2.6111) = 0.1667\). Notice that \(X_3\) alone is strongly related to \(Y\) (\(\sum x_3y = 22\), nearly as large as \(\sum x_2y\)), yet its coefficient is small once \(X_2\) is in the equation, because \(X_2\) and \(X_3\) are themselves highly correlated. Multiple regression credits each variable only with what it adds beyond the others.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.