4.2 Scaled residuals
Raw residuals are in the units of \(y\), which makes it hard to say whether a residual is “large”. Scaling puts them on a common footing.
Definition 4.2.1. The standardized residuals are \[ d_i = \frac {e_i}{\sqrt {MS_{\mathrm {Res}}}} , \qquad i = 1, \dots , n . \] The studentized residuals are \[ r_i = \frac {e_i}{\sqrt {MS_{\mathrm {Res}}(1 - h_{ii})}} , \qquad i = 1, \dots , n . \]
The standardized residuals have mean zero and approximately unit variance; a \(d_i\) larger than \(3\) in size suggests an outlier. But \(d_i\) divides every residual by the same estimate of \(\sigma \), while Theorem 4.1.1 says that \(e_i\) has standard deviation \(\sigma \sqrt {1 - h_{ii}}\). The studentized residual divides by the right thing, and so has variance close to \(1\) whatever the leverage. When \(n\) is large and no point has high leverage the two are almost equal; when they differ, \(r_i\) is the one to trust, because the points where they differ most are exactly the high-leverage points whose raw residuals understate how badly they fit.
Deleting one observation
The most revealing way to judge whether observation \(i\) fits is to fit the model without it, and see how well the result predicts \(y_i\). Doing this by refitting \(n\) times would be tedious; the remarkable fact is that it is never necessary. Everything about the deleted fit can be read off the full fit. Write \(\mathbf {x}_i^\top \) for row \(i\) of \(\mathbf {X}\), \(\mathbf {X}_{(i)}\) and \(\mathbf {y}_{(i)}\) for \(\mathbf {X}\) and \(\mathbf {y}\) with row \(i\) removed, and \(\hat {\boldsymbol \beta }_{(i)}\) and \(S^2_{(i)}\) for the estimates of \(\boldsymbol \beta \) and \(\sigma ^2\) computed from the remaining \(n - 1\) observations.
Lemma 4.2.2 (Sherman–Morrison). Let \(\mathbf {A}\) be an invertible \(p \times p\) matrix and \(\mathbf {u}\) a vector with \(\mathbf {u}^\top \mathbf {A}^{-1}\mathbf {u} \neq 1\). Then \(\mathbf {A} - \mathbf {u}\mathbf {u}^\top \) is invertible and \[ (\mathbf {A} - \mathbf {u}\mathbf {u}^\top )^{-1} = \mathbf {A}^{-1} + \frac {\mathbf {A}^{-1}\mathbf {u}\mathbf {u}^\top \mathbf {A}^{-1}}{1 - \mathbf {u}^\top \mathbf {A}^{-1}\mathbf {u}} . \]
Proof. Write \(c = \mathbf {u}^\top \mathbf {A}^{-1}\mathbf {u}\), a number, and multiply out: \begin {align*} &(\mathbf {A} - \mathbf {u}\mathbf {u}^\top )\left [\mathbf {A}^{-1} + \frac {\mathbf {A}^{-1}\mathbf {u}\mathbf {u}^\top \mathbf {A}^{-1}}{1 - c}\right ] \\ &\quad = \mathbf {I} - \mathbf {u}\mathbf {u}^\top \mathbf {A}^{-1} + \frac {\mathbf {u}\mathbf {u}^\top \mathbf {A}^{-1} - \mathbf {u}(\mathbf {u}^\top \mathbf {A}^{-1}\mathbf {u})\mathbf {u}^\top \mathbf {A}^{-1}}{1 - c} \\ &\quad = \mathbf {I} - \mathbf {u}\mathbf {u}^\top \mathbf {A}^{-1} + \frac {(1 - c)\,\mathbf {u}\mathbf {u}^\top \mathbf {A}^{-1}}{1 - c} = \mathbf {I} . \end {align*}
A square matrix with a right inverse is invertible, and the right inverse is the inverse. □
Theorem 4.2.3 (Deletion formulas). Suppose \(h_{ii} < 1\), and write \(\mathbf {C} = (\mathbf {X}^\top \mathbf {X})^{-1}\). Then
- (i)
- \(\mathbf {x}_i^\top (\mathbf {X}_{(i)}^\top \mathbf {X}_{(i)})^{-1}\mathbf {x}_i = \dfrac {h_{ii}}{1 - h_{ii}}\);
- (ii)
- \(\hat {\boldsymbol \beta }_{(i)} = \hat {\boldsymbol \beta } - \dfrac {\mathbf {C}\mathbf {x}_i\, e_i}{1 - h_{ii}}\);
- (iii)
- the prediction error of \(y_i\) from the fit without it is \(y_i - \mathbf {x}_i^\top \hat {\boldsymbol \beta }_{(i)} = \dfrac {e_i}{1 - h_{ii}}\);
- (iv)
- the residual sum of squares without observation \(i\) is \(SS_{\mathrm {Res}} - \dfrac {e_i^2}{1 - h_{ii}}\), so that \[ S^2_{(i)} = \frac {(n - p)MS_{\mathrm {Res}} - e_i^2/(1 - h_{ii})}{n - p - 1} . \]
Proof. Deleting a row removes one term from each of the sums that make up \(\mathbf {X}^\top \mathbf {X}\) and \(\mathbf {X}^\top \mathbf {y}\): \[ \mathbf {X}_{(i)}^\top \mathbf {X}_{(i)} = \mathbf {X}^\top \mathbf {X} - \mathbf {x}_i\mathbf {x}_i^\top , \qquad \mathbf {X}_{(i)}^\top \mathbf {y}_{(i)} = \mathbf {X}^\top \mathbf {y} - \mathbf {x}_i y_i . \] Note \(\mathbf {x}_i^\top \mathbf {C}\mathbf {x}_i = h_{ii} \neq 1\), so Lemma 4.2.2 applies with \(\mathbf {A} = \mathbf {X}^\top \mathbf {X}\) and \(\mathbf {u} = \mathbf {x}_i\): \[ (\mathbf {X}_{(i)}^\top \mathbf {X}_{(i)})^{-1} = \mathbf {C} + \frac {\mathbf {C}\mathbf {x}_i\mathbf {x}_i^\top \mathbf {C}}{1 - h_{ii}} . \]
(i) Multiply on the left by \(\mathbf {x}_i^\top \) and on the right by \(\mathbf {x}_i\): \(h_{ii} + h_{ii}^2/(1 - h_{ii}) = h_{ii}/(1 - h_{ii})\).
(ii) Using \(\mathbf {C}\mathbf {X}^\top \mathbf {y} = \hat {\boldsymbol \beta }\) and \(\mathbf {x}_i^\top \hat {\boldsymbol \beta } = \hat y_i\), \begin {align*} \hat {\boldsymbol \beta }_{(i)} &= \left [\mathbf {C} + \frac {\mathbf {C}\mathbf {x}_i\mathbf {x}_i^\top \mathbf {C}}{1 - h_{ii}}\right ](\mathbf {X}^\top \mathbf {y} - \mathbf {x}_i y_i) \\ &= \hat {\boldsymbol \beta } - \mathbf {C}\mathbf {x}_i y_i + \frac {\mathbf {C}\mathbf {x}_i\,\hat y_i - \mathbf {C}\mathbf {x}_i\, h_{ii} y_i}{1 - h_{ii}} \\ &= \hat {\boldsymbol \beta } + \frac {\mathbf {C}\mathbf {x}_i}{1 - h_{ii}} \bigl [-(1 - h_{ii})y_i + \hat y_i - h_{ii}y_i\bigr ] \\ &= \hat {\boldsymbol \beta } - \frac {\mathbf {C}\mathbf {x}_i\, e_i}{1 - h_{ii}} . \end {align*}
(iii) Multiply (ii) on the left by \(\mathbf {x}_i^\top \): \(\mathbf {x}_i^\top \hat {\boldsymbol \beta }_{(i)} = \hat y_i - h_{ii}e_i/(1 - h_{ii})\), so \[ y_i - \mathbf {x}_i^\top \hat {\boldsymbol \beta }_{(i)} = e_i + \frac {h_{ii}e_i}{1 - h_{ii}} = \frac {e_i}{1 - h_{ii}} . \]
(iv) For any \(j\), multiplying (ii) by \(\mathbf {x}_j^\top \) and using \(\mathbf {x}_j^\top \mathbf {C}\mathbf {x}_i = h_{ji}\) gives \(y_j - \mathbf {x}_j^\top \hat {\boldsymbol \beta }_{(i)} = e_j + h_{ji}e_i/(1 - h_{ii})\). Square and add over all \(j\), including \(j = i\) for the moment: \[ \sum _j \Bigl (e_j + \frac {h_{ji}e_i}{1 - h_{ii}}\Bigr )^2 = \sum _j e_j^2 + \frac {2e_i}{1 - h_{ii}}\sum _j h_{ij}e_j + \frac {e_i^2}{(1 - h_{ii})^2}\sum _j h_{ij}^2 . \] Here \(\sum _j h_{ij}e_j\) is entry \(i\) of \(\mathbf {He} = \mathbf {H}(\mathbf {I} - \mathbf {H})\mathbf {y} = \mathbf {0}\), and \(\sum _j h_{ij}^2 = (\mathbf {H}^2)_{ii} = h_{ii}\). So the sum is \(SS_{\mathrm {Res}} + h_{ii}e_i^2/(1 - h_{ii})^2\). The deleted fit’s residual sum of squares leaves out the \(j = i\) term, which by (iii) is \(e_i^2/(1 - h_{ii})^2\): \[ SS_{\mathrm {Res}(i)} = SS_{\mathrm {Res}} + \frac {(h_{ii} - 1)e_i^2}{(1 - h_{ii})^2} = SS_{\mathrm {Res}} - \frac {e_i^2}{1 - h_{ii}} . \] The deleted fit has \(n - 1\) observations and \(p\) parameters, so \(S^2_{(i)} = SS_{\mathrm {Res}(i)}/(n - p - 1)\), and \(SS_{\mathrm {Res}} = (n - p)MS_{\mathrm {Res}}\). □
Part (iii) gives the first of two further residuals.
Definition 4.2.4. The PRESS residuals (deleted residuals) are \[ e_{(i)} = y_i - \hat y_{(i)} = \frac {e_i}{1 - h_{ii}} , \qquad i = 1, \dots , n , \] where \(\hat y_{(i)} = \mathbf {x}_i^\top \hat {\boldsymbol \beta }_{(i)}\) is the prediction of \(y_i\) from the other \(n - 1\) observations. The externally studentized residuals, or R-student residuals, are \[ t_i = \frac {e_i}{\sqrt {S^2_{(i)}(1 - h_{ii})}} , \qquad i = 1, \dots , n . \]
The studentized residual \(r_i\) estimates \(\sigma ^2\) by \(MS_{\mathrm {Res}}\), which includes observation \(i\) itself; this is called internal scaling. If observation \(i\) is badly out of line, it inflates \(MS_{\mathrm {Res}}\) and so shrinks its own \(r_i\). R-student uses the estimate \(S^2_{(i)}\) from which observation \(i\) has been removed, so a wild point cannot hide behind the variance it has inflated.
- (i)
- Under A1–A3, \(E(e_{(i)}) = 0\) and \(\operatorname {Var}(e_{(i)}) = \sigma ^2/(1 - h_{ii})\). Divided by its standard deviation, the PRESS residual becomes \(e_i\big /\sqrt {\sigma ^2(1 - h_{ii})}\), the ordinary residual divided by its own.
- (ii)
- Under A1–A4, \(t_i \sim t_{n - p - 1}\).
- (iii)
- \(r_i^2 \le n - p\), and \[ t_i = r_i\sqrt {\frac {n - p - 1}{n - p - r_i^2}} , \] so \(t_i\) is an increasing function of \(r_i\), and \(|t_i| > |r_i|\) exactly when \(|r_i| > 1\).
Proof. (i) \(e_{(i)} = e_i/(1 - h_{ii})\), and by Theorem 4.1.1 \(e_i\) has mean \(0\) and variance \(\sigma ^2(1 - h_{ii})\); divide the variance by \((1 - h_{ii})^2\).
(ii) The deleted fit uses only \(\mathbf {y}_{(i)}\), which is independent of \(y_i\) by A3 and A4. Applied to the deleted data, Theorem 3.7.2 gives \(\hat {\boldsymbol \beta }_{(i)} \sim N_p\bigl (\boldsymbol \beta , \sigma ^2(\mathbf {X}_{(i)}^\top \mathbf {X}_{(i)})^{-1}\bigr )\) and \((n - p - 1)S^2_{(i)}/\sigma ^2 \sim \chi ^2_{n - p - 1}\), the two independent. So \(e_{(i)} = y_i - \mathbf {x}_i^\top \hat {\boldsymbol \beta }_{(i)}\) is a difference of independent normal quantities, normal with mean \(\mathbf {x}_i^\top \boldsymbol \beta - \mathbf {x}_i^\top \boldsymbol \beta = 0\) and, by Theorem 4.2.3(i), variance \[ \sigma ^2 + \sigma ^2\,\mathbf {x}_i^\top (\mathbf {X}_{(i)}^\top \mathbf {X}_{(i)})^{-1}\mathbf {x}_i = \sigma ^2\left (1 + \frac {h_{ii}}{1 - h_{ii}}\right ) = \frac {\sigma ^2}{1 - h_{ii}} . \] Both \(y_i\) and \(\hat {\boldsymbol \beta }_{(i)}\) are independent of \(S^2_{(i)}\), hence so is \(e_{(i)}\). By (F3), \[ \frac {e_{(i)}\sqrt {1 - h_{ii}}/\sigma }{\sqrt {S^2_{(i)}/\sigma ^2}} = \frac {e_{(i)}\sqrt {1 - h_{ii}}}{S_{(i)}} \sim t_{n - p - 1} , \] and since \(e_{(i)}\sqrt {1 - h_{ii}} = e_i/\sqrt {1 - h_{ii}}\), the left side is \(t_i\).
(iii) The deleted residual sum of squares in Theorem 4.2.3(iv) cannot be negative, so \(e_i^2/(1 - h_{ii}) \le SS_{\mathrm {Res}} = (n - p)MS_{\mathrm {Res}}\), which is \(r_i^2 \le n - p\). Dividing that same formula by \(MS_{\mathrm {Res}}\), \[ \frac {(n - p - 1)S^2_{(i)}}{MS_{\mathrm {Res}}} = n - p - r_i^2 , \] and so \[ t_i^2 = \frac {e_i^2}{(1 - h_{ii})MS_{\mathrm {Res}}}\cdot \frac {MS_{\mathrm {Res}}}{S^2_{(i)}} = r_i^2\,\frac {n - p - 1}{n - p - r_i^2} . \] \(t_i\) and \(r_i\) have the sign of \(e_i\), which gives the formula. The factor \((n - p - 1)/(n - p - r_i^2)\) exceeds \(1\) exactly when \(r_i^2 > 1\). □
Part (iii) is a warning about studentized residuals in small samples: they are bounded. With \(n - p = 3\), no studentized residual can exceed \(\sqrt 3 = 1.73\) in size however wild the observation, so the rule “\(|r_i| > 3\) is suspicious” can never fire. R-student has no such ceiling, and part (ii) gives it an exact reference distribution. For these reasons R-student is the residual of choice for detecting outliers.
Example 4.2.6. Compute the four kinds of scaled residual for the crop data of Example 3.2.1.
Solution. From Examples 3.4.3 and 3.6.2, \(MS_{\mathrm {Res}} = 4/3\) with \(n - p = 3\), the residuals are \(-1, 1, 1, -1, 0, 0\), and the leverages are \(h_{ii} = \mathbf {x}_i^\top (\mathbf {X}^\top \mathbf {X})^{-1}\mathbf {x}_i\). Exactly, with the adjugate of Example 3.3.3: for plot 1, \(\mathbf {x}_1 = (1, 1, 2)^\top \) and \[ \mathbf {x}_1^\top \begin {bmatrix} 510 & -24 & -150 \\ -24 & 33 & -30 \\ -150 & -30 & 96 \end {bmatrix} \mathbf {x}_1 = 159 , \qquad h_{11} = \frac {159}{378} = \frac {53}{126} = 0.4206 . \] The others are found the same way. For plot 1, \begin {align*} d_1 &= \frac {-1}{\sqrt {4/3}} = -0.866, \\ r_1 &= \frac {-1}{\sqrt {(4/3)(0.5794)}} = -1.138, \\ e_{(1)} &= \frac {-1}{0.5794} = -1.726, \\ S^2_{(1)} &= \frac {4 - (-1)^2/0.5794}{2} = \frac {4 - 1.726}{2} = 1.137, \\ t_1 &= \frac {-1}{\sqrt {1.137 \times 0.5794}} = -1.232 . \end {align*}
In full:
| Plot | \(e_i\) | \(h_{ii}\) | \(d_i\) | \(r_i\) | \(e_{(i)}\) | \(S^2_{(i)}\) | \(t_i\) |
| 1 | \(-1\) | 0.421 | \(-0.866\) | \(-1.138\) | \(-1.726\) | 1.137 | \(-1.232\) |
| 2 | \(1\) | 0.587 | \(0.866\) | \(1.348\) | \(2.423\) | 0.788 | \(1.753\) |
| 3 | \(1\) | 0.397 | \(0.866\) | \(1.115\) | \(1.658\) | 1.171 | \(1.190\) |
| 4 | \(-1\) | 0.230 | \(-0.866\) | \(-0.987\) | \(-1.299\) | 1.351 | \(-0.981\) |
| 5 | \(0\) | 0.587 | \(0\) | \(0\) | \(0\) | 2.000 | \(0\) |
| 6 | \(0\) | 0.778 | \(0\) | \(0\) | \(0\) | 2.000 | \(0\) |
The four non-zero raw residuals are equal in size, yet plot 2, with leverage \(0.587\), has the largest scaled residual and plot 4, with leverage \(0.230\), the smallest: the same miss counts for more at a point that pulls the plane towards itself. Check on (iii) of Theorem 4.2.5 for plot 2: \(r_2^2 = 1.817\) and \(1.348\sqrt {2/(3 - 1.817)} = 1.348 \times 1.300 = 1.753 = t_2\). Every \(|r_i|\) is below \(\sqrt 3\), as it must be.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.