4.8 Leverage, Outliers and Influential Observations
Residual plots find observations that the model fits badly. They do not find the observations that determined the fit in the first place, and those are the more dangerous: a point can sit almost exactly on the fitted surface precisely because it dragged the surface to itself. Such a point has a small residual and a large effect, so no examination of residuals alone will reveal it.
Three questions must be separated. Is the observation extreme in the predictors? Is it fitted badly? Would removing it change the conclusions? These are leverage, outlyingness and influence, and an observation may have any combination of them.
Definition 4.8.1 (Leverage). The leverage of observation \(i\) is \(h_{ii}\), the \(i^{\text {th}}\) diagonal element of the hat matrix \(H=X\left (X^tX\right )^{-1}X^t\).
Result 4.8.2. The leverages satisfy \[0\leq h_{ii}\leq 1,\qquad \sum _{i=1}^{n}h_{ii}=tr(H)=k,\] so the average leverage is \(k/n\). An observation is conventionally called a high leverage point when \(h_{ii}>2k/n\).
Note. Leverage is computed from \(X\) alone — the response never enters it. It measures how far observation \(i\) is from the centre of the predictor space, and \(h_{ii}\) is exactly the weight that \(Y_i\) receives in its own fitted value, since \(\widehat {Y}_i=\sum _j h_{ij}Y_j\). A point with \(h_{ii}\) near \(1\) fits itself: \(\widehat {Y}_i\approx Y_i\) whatever \(Y_i\) happens to be. This is why \(\text {Var}\left (\widehat {e}_i\right )=\sigma ^2\left (1-h_{ii}\right )\) is small for high leverage points, and why dividing by \(\sqrt {1-h_{ii}}\) is necessary before residuals can be compared with one another at all.
Definition 4.8.3 (Studentised residuals). The internally studentised residual is \[r_i=\frac {\widehat {e}_i}{s\sqrt {1-h_{ii}}},\qquad s^2=MSE,\] and the externally studentised (or deleted) residual is \[t_i=\frac {\widehat {e}_i}{s_{(i)}\sqrt {1-h_{ii}}},\] where \(s_{(i)}\) is computed from the fit with observation \(i\) omitted.
Note. The distinction matters. A genuine outlier inflates \(s\), so it inflates its own denominator and disguises itself — \(r_i\) can never exceed \(\sqrt {n-k}\) no matter how aberrant the point. In \(t_i\) the offending observation is excluded from the scale estimate, so it cannot mask itself, and under the model \(t_i\sim t_{n-k-1}\) exactly, which makes a formal test possible. When the two disagree, trust \(t_i\).
Definition 4.8.4 (Cook’s distance). \[D_i=\frac {\left (\widehat {\underline {B}}-\widehat {\underline {B}}_{(i)}\right )^t X^tX\left (\widehat {\underline {B}}-\widehat {\underline {B}}_{(i)}\right )}{k\,s^2} =\frac {r_i^{2}}{k}\cdot \frac {h_{ii}}{1-h_{ii}},\] where \(\widehat {\underline {B}}_{(i)}\) is the coefficient vector fitted without observation \(i\).
Note. The second expression is the one to remember, because it displays influence as a product of the two separate things: \(r_i^2\) measures how badly the point is fitted, and \(h_{ii}/(1-h_{ii})\) measures how extreme it is in the predictors. Influence requires both. A wild \(Y\) at the centre of the design has a large \(r_i\) but tiny leverage and moves nothing; a point far out in \(X\) that happens to lie on the true line has large leverage and a small residual and also moves nothing. It is the combination that is dangerous.
Values of \(D_i\) above \(1\) are usually worth investigating, though the comparison to make is between the \(D_i\) of a given data set rather than against any fixed threshold.
Note (What to do about it). Finding an influential observation is not grounds for deleting it. The question to ask is why it is influential. If the record is a transcription error, correct it. If it is a genuine observation from a different regime, the model is wrong over the range that includes it, and that is a finding to report, not a point to discard. If it is genuine and in range, it stays — and the honest report gives the fit with and without it, so the reader can see how much of the conclusion rests on one row of the data.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.