8.3 Diagnostics
The correlation matrix
Large pairwise correlations \(|r_{jl}|\) signal a near dependence between two regressors. But a near dependence among three or more regressors can exist when every pairwise correlation is modest, so the correlation matrix can miss it (Example 8.3.2).
Variance inflation factors
The VIFs are the diagonal of \(\mathbf {R}^{-1}\), and Theorem 8.2.1 makes them easy to interpret. The usual rule is that a VIF above \(5\) deserves attention and one above \(10\) (that is, \(R_j^2 > 0.9\)) indicates that \(\beta _j\) is poorly estimated because of multicollinearity. VIFs detect dependences of any number of regressors, but they do not say how many dependences there are or which regressors are involved.
Eigenvalues and the condition number
Theorem 8.3.1. If \(\mathbf {R}\mathbf {v} = \lambda \mathbf {v}\) with \(\|\mathbf {v}\| = 1\), then \[ \sum _{i=1}^n\Bigl (\sum _{j=1}^k v_jw_{ij}\Bigr )^2 = \lambda . \] So if \(\lambda \) is small, \(\sum _j v_jw_{ij} \approx 0\) for every observation \(i\): the columns of \(\mathbf {W}\) are nearly dependent, with coefficients given by \(\mathbf {v}\).
Proof. The left side is \(\|\mathbf {Wv}\|^2 = \mathbf {v}^\top \mathbf {W}^\top \mathbf {W}\mathbf {v} = \mathbf {v}^\top \mathbf {R}\mathbf {v} = \lambda \mathbf {v}^\top \mathbf {v} = \lambda \). □
So each small eigenvalue signals one near dependence, and its eigenvector says which regressors are involved: those with large entries \(v_j\). Because the eigenvalues add to \(k\), a small one is small relative to the others, and the standard summary is the condition number \[ \kappa = \frac {\lambda _{\max }}{\lambda _{\min }} . \] As a rough guide, \(\kappa < 100\) indicates no serious problem, \(100 \le \kappa \le 1000\) moderate to strong multicollinearity, and \(\kappa > 1000\) severe multicollinearity. The condition indices \(\lambda _{\max }/\lambda _j\) apply the same scale to each eigenvalue, and the number of large indices is the number of near dependences. For the appliance data, \(\kappa = 1.9856/0.0144 = 138\).
Example 8.3.2. Ten observations on three regressors:
| \(x_1\) | 7 | 7 | 6 | 6 | 1 | 6 | 7 | 2 | 2 | 1 |
| \(x_2\) | 6 | 7 | 5 | 5 | 6 | 1 | 6 | 7 | 1 | 7 |
| \(x_3\) | 14 | 13 | 11 | 12 | 8 | 6 | 13 | 9 | 3 | 9 |
have correlation matrix, inverse diagonal and eigenvalues \[ \mathbf {R} = \begin {bmatrix} 1 & -0.028 & 0.653 \\ -0.028 & 1 & 0.714 \\ 0.653 & 0.714 & 1 \end {bmatrix} , \] VIFs \(13.0\), \(15.2\) and \(26.5\), and eigenvalues \(1.954\), \(1.028\) and \(0.0188\). Interpret.
Solution. No pairwise correlation exceeds \(0.72\), and \(x_1\) and \(x_2\) are uncorrelated; by the correlation matrix alone there is no problem. But all three VIFs exceed \(10\): each regressor is very well explained by the other two (\(R_j^2 = 0.923\), \(0.934\), \(0.962\)). There is one small eigenvalue, with condition number \(\kappa = 1.954/0.0188 = 104\), so there is one near dependence of moderate strength. Its eigenvector is \(\mathbf {v} = (0.482,\ 0.525,\ -0.702)\), so by Theorem 8.3.1 \[ 0.482\,w_1 + 0.525\,w_2 - 0.702\,w_3 \approx 0 . \] Converting back with \(w_j = (x_j - \bar x_j)/\sqrt {S_{jj}}\), where \(S_{11} = 62.5\), \(S_{22} = 46.9\), \(S_{33} = 109.6\), the coefficients become \(0.482/\sqrt {62.5} = 0.0609\), \(0.525/\sqrt {46.9} = 0.0766\) and \(-0.702/\sqrt {109.6} = -0.0671\), so that \[ x_3 \approx \text {constant} + 0.91x_1 + 1.14x_2 . \] Indeed in these data \(x_3\) differs from \(x_1 + x_2\) by at most \(1\). The three regressors carry the information of two, and a model containing all three cannot estimate their separate coefficients well. The correlation matrix missed this because the dependence involves three regressors at once.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.