5.1 Leverage
The leverages \(h_{ii}\) are the diagonal entries of the hat matrix. By Theorem 3.4.2, \(0 \le h_{ii} \le 1\) and \(\sum h_{ii} = p\), so the average leverage is \(p/n\). The next result shows what \(h_{ii}\) measures.
Theorem 5.1.1. Suppose the model has an intercept. Let \(\mathbf {Z}\) be the \(n \times k\) matrix of regressor values with each column centred at its mean, and let \(\mathbf {z}_i^\top \) be its row \(i\), the vector of deviations \(x_{ij} - \bar x_j\). Then \[ h_{ii} = \frac {1}{n} + \mathbf {z}_i^\top (\mathbf {Z}^\top \mathbf {Z})^{-1}\mathbf {z}_i . \] In particular \(h_{ii} \ge 1/n\), with equality exactly when \(\mathbf {x}_i\) is at the centroid \((\bar x_1, \dots , \bar x_k)\).
Proof. Subtracting \(\bar x_j\) times the column of ones from column \(j\) of \(\mathbf {X}\) does not change the column space, so the matrix \(\mathbf {W} = [\mathbf {1} \;\; \mathbf {Z}]\) has the same column space as \(\mathbf {X}\). Projection onto a subspace does not depend on which basis is used to describe it, so \(\mathbf {H}\) is also the hat matrix of \(\mathbf {W}\). (Directly: \(\mathbf {W} = \mathbf {XA}\) for an invertible \(\mathbf {A}\), and \(\mathbf {W}(\mathbf {W}^\top \mathbf {W})^{-1}\mathbf {W}^\top = \mathbf {XA}\mathbf {A}^{-1}(\mathbf {X}^\top \mathbf {X})^{-1}(\mathbf {A}^\top )^{-1}\mathbf {A}^\top \mathbf {X}^\top = \mathbf {H}\).) The columns of \(\mathbf {Z}\) add to zero, so \(\mathbf {1}^\top \mathbf {Z} = \mathbf {0}^\top \) and \[ \mathbf {W}^\top \mathbf {W} = \begin {bmatrix} n & \mathbf {0}^\top \\ \mathbf {0} & \mathbf {Z}^\top \mathbf {Z} \end {bmatrix}, \qquad \mathbf {H} = \frac {1}{n}\mathbf {1}\mathbf {1}^\top + \mathbf {Z}(\mathbf {Z}^\top \mathbf {Z})^{-1}\mathbf {Z}^\top . \] Take the \((i,i)\) entry. \(\mathbf {Z}^\top \mathbf {Z}\) is positive definite (it is invertible, and \(\mathbf {v}^\top \mathbf {Z}^\top \mathbf {Z}\mathbf {v} = \|\mathbf {Zv}\|^2\)), and so is its inverse, so the second term is \(\ge 0\) and is zero only when \(\mathbf {z}_i = \mathbf {0}\). □
So \(h_{ii}\) is \(1/n\) plus a measure of the distance of \(\mathbf {x}_i\) from the centre of the data, a distance that takes account of the spread of each regressor and of the correlation between them. With one regressor, \(\mathbf {Z}^\top \mathbf {Z} = S_{xx}\) and we recover \(h_{ii} = 1/n + (x_i - \bar x)^2/S_{xx}\) (Example 4.1.2). With two or more regressors, the points of equal leverage lie on ellipses (ellipsoids) centred at the centroid, so a point can have high leverage even when each of its coordinates, taken alone, is ordinary: it is the combination that is unusual.
The usual rule calls observation \(i\) a high-leverage point if \[ h_{ii} > \frac {2p}{n} , \] twice the average. The rule needs \(2p/n < 1\) to be of use, and is best used to pick out points for a closer look rather than as a test.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.