5.2 Cook’s distance
The most direct measure of influence asks how far the estimates move when observation \(i\) is deleted. A natural scale for movements of \(\hat {\boldsymbol \beta }\) comes from its confidence region.
Lemma 5.2.1 (Joint confidence region). Under A1–A4, \[ \frac {(\hat {\boldsymbol \beta } - \boldsymbol \beta )^\top \mathbf {X}^\top \mathbf {X}(\hat {\boldsymbol \beta } - \boldsymbol \beta )}{p\,MS_{\mathrm {Res}}} \sim F_{p,\, n - p} , \] so the set of \(\boldsymbol \beta \) for which this quantity is at most \(F_{\alpha , p, n - p}\) is a \(100(1 - \alpha )\%\) confidence region for \(\boldsymbol \beta \). It is an ellipsoid centred at \(\hat {\boldsymbol \beta }\).
Proof. \(\hat {\boldsymbol \beta } - \boldsymbol \beta = \mathbf {C}\mathbf {X}^\top \boldsymbol \varepsilon \), so \(\mathbf {X}(\hat {\boldsymbol \beta } - \boldsymbol \beta ) = \mathbf {H}\boldsymbol \varepsilon \) and the numerator is \(\boldsymbol \varepsilon ^\top \mathbf {H}^\top \mathbf {H}\boldsymbol \varepsilon = \boldsymbol \varepsilon ^\top \mathbf {H}\boldsymbol \varepsilon \). \(\mathbf {H}\) is symmetric and idempotent with trace \(p\), and \(\mathbf {H}(\mathbf {I} - \mathbf {H}) = \mathbf {0}\), so by Lemma 3.7.1 the numerator over \(\sigma ^2\) is \(\chi ^2_p\), independent of \(SS_{\mathrm {Res}}/\sigma ^2 = \boldsymbol \varepsilon ^\top (\mathbf {I} - \mathbf {H})\boldsymbol \varepsilon /\sigma ^2 \sim \chi ^2_{n - p}\). By (F3) the ratio of each to its degrees of freedom is \(F_{p, n - p}\). □
Definition 5.2.2. Cook’s distance for observation \(i\) is \[ D_i = \frac {(\hat {\boldsymbol \beta }_{(i)} - \hat {\boldsymbol \beta })^\top \mathbf {X}^\top \mathbf {X}(\hat {\boldsymbol \beta }_{(i)} - \hat {\boldsymbol \beta })}{p\,MS_{\mathrm {Res}}} , \qquad i = 1, \dots , n . \]
\(D_i\) is the quantity of Lemma 5.2.1 with \(\hat {\boldsymbol \beta }_{(i)}\) in place of \(\boldsymbol \beta \): it says which confidence ellipsoid about \(\hat {\boldsymbol \beta }\) the deleted estimate lands on. (It is not itself \(F\)-distributed, since \(\hat {\boldsymbol \beta }_{(i)}\) is not a fixed parameter; the comparison is a yardstick, not a test.) The median of \(F_{p, n - p}\) is close to \(1\) for most \(p\) and \(n\), so \(D_i = 1\) means that deleting the point moves the estimate to the edge of a \(50\%\) confidence region, a large move. The usual rule is to regard points with \[ D_i > 1 \] as influential.
Theorem 5.2.3. Suppose \(h_{ii} < 1\). Then \[ D_i = \frac {r_i^2}{p}\cdot \frac {h_{ii}}{1 - h_{ii}} = \frac {\|\hat {\mathbf {y}} - \hat {\mathbf {y}}_{(i)}\|^2}{p\,MS_{\mathrm {Res}}} , \] where \(\hat {\mathbf {y}}_{(i)} = \mathbf {X}\hat {\boldsymbol \beta }_{(i)}\) is the vector of all \(n\) fitted values from the fit without observation \(i\).
Proof. The second form holds because \(\mathbf {X}(\hat {\boldsymbol \beta } - \hat {\boldsymbol \beta }_{(i)}) = \hat {\mathbf {y}} - \hat {\mathbf {y}}_{(i)}\) and \(\mathbf {v}^\top \mathbf {X}^\top \mathbf {X}\mathbf {v} = \|\mathbf {Xv}\|^2\). For the first, by Theorem 4.2.3(ii), \(\hat {\boldsymbol \beta } - \hat {\boldsymbol \beta }_{(i)} = \mathbf {C}\mathbf {x}_i e_i/(1 - h_{ii})\), so, using \(\mathbf {C}\mathbf {X}^\top \mathbf {X}\mathbf {C} = \mathbf {C}\) and \(\mathbf {x}_i^\top \mathbf {C}\mathbf {x}_i = h_{ii}\), \[ (\hat {\boldsymbol \beta } - \hat {\boldsymbol \beta }_{(i)})^\top \mathbf {X}^\top \mathbf {X}(\hat {\boldsymbol \beta } - \hat {\boldsymbol \beta }_{(i)}) = \frac {e_i^2\,\mathbf {x}_i^\top \mathbf {C}\mathbf {x}_i}{(1 - h_{ii})^2} = \frac {e_i^2 h_{ii}}{(1 - h_{ii})^2} . \] Divide by \(p\,MS_{\mathrm {Res}}\) and recall \(r_i^2 = e_i^2/\bigl (MS_{\mathrm {Res}}(1 - h_{ii})\bigr )\): \[ D_i = \frac {e_i^2}{MS_{\mathrm {Res}}(1 - h_{ii})}\cdot \frac {h_{ii}}{p(1 - h_{ii})} = \frac {r_i^2}{p}\cdot \frac {h_{ii}}{1 - h_{ii}} . \qedhere \] □
The theorem is the formal statement of the three panels of Figure 10. Cook’s distance is a product of two factors: \(r_i^2\), which measures how badly the point fits, and \(h_{ii}/(1 - h_{ii})\), which grows without bound as \(h_{ii} \to 1\) and measures how much leverage the point has. A large \(D_i\) needs both, or an extreme amount of one. A point with zero residual has \(D_i = 0\) whatever its leverage; a point with leverage \(1/n\) at the centroid has \(h_{ii}/(1 - h_{ii}) = 1/(n - 1)\) and so can have a large \(D_i\) only with a very large residual.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.