4.1 Residuals and their properties
Recall that \(\mathbf {e} = \mathbf {y} - \hat {\mathbf {y}} = (\mathbf {I} - \mathbf {H})\mathbf {y}\), where \(\mathbf {H} = \mathbf {X}(\mathbf {X}^\top \mathbf {X})^{-1}\mathbf {X}^\top \) is the hat matrix and \(h_{ii}\) its diagonal entries.
- (i)
- \(E(\mathbf {e}) = \mathbf {0}\);
- (ii)
- \(\operatorname {Var}(\mathbf {e}) = \sigma ^2(\mathbf {I} - \mathbf {H})\), so \(\operatorname {Var}(e_i) = \sigma ^2(1 - h_{ii})\) and \(\operatorname {Cov}(e_i, e_j) = -\sigma ^2 h_{ij}\) for \(i \neq j\);
- (iii)
- \(\operatorname {Cov}(\mathbf {e}, \hat {\mathbf {y}}) = \mathbf {0}\).
Proof. Since \((\mathbf {I} - \mathbf {H})\mathbf {X} = \mathbf {0}\) (Theorem 3.4.2), \(\mathbf {e} = (\mathbf {I} - \mathbf {H})(\mathbf {X}\boldsymbol \beta + \boldsymbol \varepsilon ) = (\mathbf {I} - \mathbf {H})\boldsymbol \varepsilon \).
(i) \(E(\mathbf {e}) = (\mathbf {I} - \mathbf {H})E(\boldsymbol \varepsilon ) = \mathbf {0}\) by A1.
(ii) By Lemma 3.1.3 and the symmetry and idempotence of \(\mathbf {I} - \mathbf {H}\), \[ \operatorname {Var}(\mathbf {e}) = (\mathbf {I} - \mathbf {H})(\sigma ^2\mathbf {I})(\mathbf {I} - \mathbf {H})^\top = \sigma ^2(\mathbf {I} - \mathbf {H})^2 = \sigma ^2(\mathbf {I} - \mathbf {H}) . \] The diagonal and off-diagonal entries give the rest.
(iii) \(\hat {\mathbf {y}} = \mathbf {Hy}\), so \(\operatorname {Cov}(\mathbf {e}, \hat {\mathbf {y}}) = (\mathbf {I} - \mathbf {H})(\sigma ^2\mathbf {I})\mathbf {H}^\top = \sigma ^2(\mathbf {H} - \mathbf {H}^2) = \mathbf {0}\). □
Two consequences are worth stating plainly. First, the residuals are not independent and do not have equal variances, even when the errors do. The dependence is mild when \(n\) is large compared with \(p\), because the \(h_{ij}\) are then small, but the unequal variances matter: a point with high leverage \(h_{ii}\) has a residual with small variance. The fitted surface is pulled towards such a point, so its residual tends to be small whether or not the point fits. Second, the residuals carry only \(n - p\) degrees of freedom, since they satisfy the \(p\) constraints \(\mathbf {X}^\top \mathbf {e} = \mathbf {0}\).
Example 4.1.2. In simple linear regression, \[ h_{ii} = \frac {1}{n} + \frac {(x_i - \bar x)^2}{S_{xx}} , \qquad \operatorname {Var}(e_i) = \sigma ^2\left [1 - \frac {1}{n} - \frac {(x_i - \bar x)^2}{S_{xx}}\right ] . \]
Solution. By Example 3.3.2, with \(\mathbf {x}_i = (1, x_i)^\top \), \[ h_{ii} = \mathbf {x}_i^\top (\mathbf {X}^\top \mathbf {X})^{-1}\mathbf {x}_i = \frac {\sum x_j^2 - 2x_i\sum x_j + n x_i^2}{nS_{xx}} . \] Write \(\sum x_j^2 = S_{xx} + n\bar x^2\) and \(\sum x_j = n\bar x\). The numerator becomes \(S_{xx} + n(\bar x^2 - 2x_i\bar x + x_i^2) = S_{xx} + n(x_i - \bar x)^2\), and dividing by \(nS_{xx}\) gives the formula. The variance follows from Theorem 4.1.1.
So in simple regression the residuals at the two ends of the \(x\)-range have the smallest variance, and the residuals near \(\bar x\) the largest.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.