8.2 Effects of multicollinearity
Theorem 8.2.1 (Variance inflation). Let \(R_j^2\) be the coefficient of determination from regressing \(x_j\) on the other \(k - 1\) regressors (with an intercept). Then the \(j\)th diagonal entry of \(\mathbf {R}^{-1}\) is \[ \mathrm {VIF}_j = \frac {1}{1 - R_j^2} , \] and the variance of the least-squares estimate of \(\beta _j\) in the original model is \[ \operatorname {Var}(\hat \beta _j) = \frac {\sigma ^2}{S_{jj}}\cdot \frac {1}{1 - R_j^2} . \]
Proof. Apply Theorem 3.11.4 to the regression on \(\mathbf {W}\) (with intercept), taking \(w_j\) as the added column. Then \((\mathbf {R}^{-1})_{jj} = 1/\mathbf {u}^\top \mathbf {u}\), where \(\mathbf {u}\) is the vector of residuals from regressing \(w_j\) on the other columns and the intercept. \(\mathbf {u}^\top \mathbf {u}\) is that regression’s residual sum of squares, which is \((1 - R_j^2)\) times its total sum of squares; and the total sum of squares of \(w_j\) is \(1\). Scaling a regressor does not change \(R_j^2\), so \(R_j^2\) is the same whether computed from \(x_j\) or \(w_j\).
For the variance, \(\hat \beta _j = \hat b_j\sqrt {SS_T/S_{jj}}\) and \(\operatorname {Var}(\hat b_j) = (\sigma ^2/SS_T)\,\mathrm {VIF}_j\), so \[ \operatorname {Var}(\hat \beta _j) = \frac {SS_T}{S_{jj}}\cdot \frac {\sigma ^2}{SS_T}\,\mathrm {VIF}_j = \frac {\sigma ^2}{S_{jj}}\,\mathrm {VIF}_j . \qedhere \] □
If \(x_j\) were uncorrelated with the other regressors, \(R_j^2 = 0\) and its variance would be \(\sigma ^2/S_{jj}\), the same as in a simple regression on \(x_j\) alone. The factor \(1/(1 - R_j^2)\) is how many times larger the variance is because of \(x_j\)’s relationship with the others: hence the name variance inflation factor. With \(R_j^2 = 0.9\) it is \(10\); with \(R_j^2 = 0.99\), \(100\), and the standard error is ten times what it would be in an orthogonal design. This is Theorem 3.11.4 again: only the part of \(x_j\) not explained by the others carries information about \(\beta _j\), and when \(R_j^2\) is near \(1\) that part is tiny.
The effect on the estimates as a whole is seen through the eigenvalues \(\lambda _1 \ge \lambda _2 \ge \cdots \ge \lambda _k\) of \(\mathbf {R}\). These are positive (since \(\mathbf {R} = \mathbf {W}^\top \mathbf {W}\) is positive definite) and add to \(\operatorname {tr}(\mathbf {R}) = k\).
Theorem 8.2.2. The squared distance between \(\hat {\mathbf {b}}\) and \(\mathbf {b}\) has expectation \[ E\|\hat {\mathbf {b}} - \mathbf {b}\|^2 = \sigma _*^2\sum _{j=1}^k\frac {1}{\lambda _j} , \qquad \text {and}\qquad E\|\hat {\mathbf {b}}\|^2 = \|\mathbf {b}\|^2 + \sigma _*^2\sum _{j=1}^k\frac {1}{\lambda _j} . \]
Proof. \(E\|\hat {\mathbf {b}} - \mathbf {b}\|^2 = \sum _j E(\hat b_j - b_j)^2 = \sum _j\operatorname {Var}(\hat b_j) = \sigma _*^2\operatorname {tr}(\mathbf {R}^{-1})\), since \(\hat {\mathbf {b}}\) is unbiased. By the spectral theorem \(\mathbf {R} = \mathbf {P}\boldsymbol \Lambda \mathbf {P}^\top \), so \(\mathbf {R}^{-1} = \mathbf {P}\boldsymbol \Lambda ^{-1}\mathbf {P}^\top \) and by Lemma 3.1.1 \(\operatorname {tr}(\mathbf {R}^{-1}) = \operatorname {tr}(\boldsymbol \Lambda ^{-1}) = \sum 1/\lambda _j\). For the second, expand \(\|\hat {\mathbf {b}}\|^2 = \|\mathbf {b} + (\hat {\mathbf {b}} - \mathbf {b})\|^2\); the cross term has expectation \(0\). □
A single small eigenvalue is enough to make \(\sum 1/\lambda _j\) large. Then the estimate is, on average, far from the truth, and in particular too long: the least-squares coefficients tend to be too large in absolute value, often with some of them of the wrong sign, offsetting each other.
The symptoms in the computer output are characteristic:
- the overall \(F\)-test is highly significant, but none or few of the individual \(t\)-tests are;
- coefficients have signs or sizes that contradict subject-matter knowledge, or the simple correlations with \(y\);
- the estimates change greatly when a regressor or a few observations are added or removed;
- the standard errors are large.
These are clues, not proof; the diagnostics of the next subsection measure the problem directly.
Remark. Multicollinearity damages estimates of the individual coefficients. It need not damage prediction: within the region where the data lie, along the line or plane of the near dependence, the fitted values are estimated as well as ever. The danger is in predicting at a point that breaks the pattern of the data, where the poorly determined combinations of coefficients come into play.
Example 8.2.3. The monthly electricity use \(y\) (tens of kWh) of eight households was recorded with the number of electrical appliances \(x_1\) and the number of those used every day \(x_2\):
| Household | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| \(x_1\) | 8 | 1 | 2 | 3 | 2 | 8 | 8 | 6 |
| \(x_2\) | 7 | 0 | 1 | 3 | 2 | 8 | 7 | 5 |
| \(y\) | 16 | 8 | 5 | 7 | 6 | 17 | 15 | 12 |
Fit \(y = \beta _0 + \beta _1x_1 + \beta _2x_2 + \varepsilon \) and examine it for multicollinearity.
Solution. The fit. \(\bar x_1 = 4.75\), \(\bar x_2 = 4.125\), \(\bar y = 10.75\), and the corrected sums of squares and products are \begin {align*} S_{11} &= 65.5, & S_{22} &= 64.875, & S_{12} &= 64.25, \\ S_{1y} &= 98.5, & S_{2y} &= 96.25, & SS_T &= 163.5 . \end {align*}
With two regressors the centred normal equations are \(S_{11}\hat \beta _1 + S_{12}\hat \beta _2 = S_{1y}\) and \(S_{12}\hat \beta _1 + S_{22}\hat \beta _2 = S_{2y}\), with determinant \(D = S_{11}S_{22} - S_{12}^2 = 4249.3125 - 4128.0625 = 121.25\), so \begin {align*} \hat \beta _1 &= \frac {S_{22}S_{1y} - S_{12}S_{2y}}{D} = \frac {6390.1875 - 6184.0625}{121.25} = \frac {206.125}{121.25} = 1.7, \\ \hat \beta _2 &= \frac {S_{11}S_{2y} - S_{12}S_{1y}}{D} = \frac {6304.375 - 6328.625}{121.25} = \frac {-24.25}{121.25} = -0.2 , \end {align*}
and \(\hat \beta _0 = 10.75 - 1.7 \times 4.75 + 0.2 \times 4.125 = 3.5\): \[ \hat y = 3.5 + 1.7x_1 - 0.2x_2 . \] \(SS_R = 1.7 \times 98.5 - 0.2 \times 96.25 = 148.2\), \(SS_{\mathrm {Res}} = 15.3\), \(MS_{\mathrm {Res}} = 3.06\), \(R^2 = 0.906\), and \[ F_0 = \frac {148.2/2}{3.06} = 24.2 > F_{0.01,2,5} = 13.27 . \] The regression is highly significant. But the standard errors are \begin {align*} \operatorname {se}(\hat \beta _1) &= \sqrt {\frac {3.06 \times 64.875}{121.25}} = 1.280, \\ \operatorname {se}(\hat \beta _2) &= \sqrt {\frac {3.06 \times 65.5}{121.25}} = 1.286 , \end {align*}
so \(t_1 = 1.33\) and \(t_2 = -0.16\), neither significant against \(t_{0.025,5} = 2.571\). And \(\hat \beta _2\) is negative, saying that, with the number of appliances fixed, using more of them every day lowers electricity use. Separately, \(y\) regressed on \(x_1\) alone has slope \(98.5/65.5 = 1.50\) (\(t = 7.6\)), and on \(x_2\) alone \(96.25/64.875 = 1.48\). All the classic symptoms are present.
Diagnosis. The correlation between the regressors is \[ r_{12} = \frac {64.25}{\sqrt {65.5 \times 64.875}} = 0.9856 , \] so \(R_1^2 = R_2^2 = r_{12}^2 = 0.9715\) and \[ \mathrm {VIF}_1 = \mathrm {VIF}_2 = \frac {1}{1 - 0.9715} = 35.0 . \] The variances are \(35\) times what they would be with uncorrelated regressors; the standard errors about six times. The correlation matrix \(\mathbf {R} = \begin {bmatrix} 1 & r \\ r & 1 \end {bmatrix}\) has eigenvalues \(1 \pm r\), that is, \(\lambda _1 = 1.9856\) and \(\lambda _2 = 0.0144\), with eigenvectors \((1, 1)/\sqrt 2\) and \((1, -1)/\sqrt 2\) (check: \(\mathbf {R}(1, \pm 1)^\top = (1 \pm r)(1, \pm 1)^\top \)). The second eigenvector says that \(w_1 - w_2 \approx 0\): households with more appliances use more of them every day, so the two standardized regressors are almost equal. The data say a great deal about the effect of \(w_1 + w_2\) and almost nothing about \(w_1 - w_2\), which is exactly what the separate coefficients need.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.