5.5 Application: the residuals of a regression
Let \(Y = X\beta + U\), where \(X\) is a non-stochastic \(n \times k\) matrix of rank \(k\) and \(U \sim N(0, \sigma ^2I_n)\). The least squares estimator, derived in the next section, is \(b = (X'X)^{-1}X'Y\), and the residual vector is \(e = Y - Xb\).
Theorem 5.5.1. With \(M = I - X(X'X)^{-1}X'\):
- (i)
- \(e = MY\) and \(e = MU\);
- (ii)
- \(M\) is symmetric and idempotent with \(\rank M = n - k\);
- (iii)
- \(e'e = U'MU\);
- (iv)
- \(E(e'e) = \sigma ^2(n - k)\), so \(s^2 = e'e/(n - k)\) is unbiased for \(\sigma ^2\);
- (v)
- \(e'e/\sigma ^2 \sim \chi ^2_{n-k}\);
- (vi)
- \(b\) and \(e\) are independent, so \(b\) and \(s^2\) are independent.
Proof. (i) \(e = Y - X(X'X)^{-1}X'Y = MY\). Since \(MX = 0\), \(MY = M(X\beta + U) = MU\).
(ii) is Example 5.1.7.
(iii) \(e'e = U'M'MU = U'MU\), because \(M' = M\) and \(M^2 = M\).
(iv) By Theorem 5.2.1(iii) with mean zero, \(E(U'MU) = \tr (M\sigma ^2I) = \sigma ^2\tr M = \sigma ^2(n - k)\). This needs only \(E(U) = 0\) and \(\Var (U) = \sigma ^2I\), not normality.
(v) \(U/\sigma \sim N(0, I)\), and \(e'e/\sigma ^2 = (U/\sigma )'M(U/\sigma ) \sim \chi ^2_{n-k}\) by Theorem 5.4.3 and (ii).
(vi) \(b = \beta + (X'X)^{-1}X'U = \beta + LU\) with \(L = (X'X)^{-1}X'\), and \(LM = (X'X)^{-1}X'M = (X'X)^{-1}(MX)' = 0\). By the argument of Theorem 5.4.4(i), \(LU\) and \(MU = e\) are independent, so \(b\) is independent of \(e\), and so of \(s^2 = e'e/(n - k)\). □
For the two-variable model \(k = 2\), and (v) and (vi) are exactly the facts \((n-2)s^2/\sigma ^2 \sim \chi ^2_{n-2}\) and “\(s^2\) is independent of \((a, b)\)” that Theorem 2.4.1 assumed. The \(t\) tests of Section 2 are now fully justified, and the same argument will justify the \(t\) and \(F\) tests of the \(k\)-variable model.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.