Solution. The inverse of \(\begin {bmatrix} 1 & r \\ r & 1 \end {bmatrix}\) is its adjugate divided by its determinant \(1 - r^2\). Then \(\operatorname {Var}(\hat {\mathbf {b}}) = \sigma _*^2\mathbf {R}^{-1}\) gives \(\operatorname {Corr}(\hat b_1, \hat b_2) = (-r\sigma _*^2/(1 - r^2))\big /(\sigma _*^2/(1 - r^2)) = -r\), and since \(\hat \beta _j\) is a positive multiple of \(\hat b_j\), the same holds for \(\hat \beta _1\), \(\hat \beta _2\). In the example the correlation is \(-0.986\): the data determine \(\beta _1 + \beta _2\) (the effect of moving along the line \(x_1 \approx x_2\)) well, so an error upwards in one estimate must be offset by an error downwards in the other. That is how \(\hat \beta _1 = 1.7\) and \(\hat \beta _2 = -0.2\) arise from true values that are probably both positive: their sum, \(1.5\), is close to the simple-regression slopes \(1.50\) and \(1.48\).

Solution. (a) \(x\): \(1, \dots , 5\), \(\bar x = 3\), \(S_{xx} = 10\). \(x^2\): \(1, 4, 9, 16, 25\), mean \(11\), with deviations \(-10, -7, -2, 5, 14\), so \(S_{x^2x^2} = 100 + 49 + 4 + 25 + 196 = 374\), and \(S_{x,x^2} = -2(-10) - 1(-7) + 0 + 1(5) + 2(14) = 60\). So \(r = 60/\sqrt {10 \times 374} = 0.981\), and both VIFs are \(1/(1 - 0.9626) = 26.7\).

(b) With \(u = x - 3\): \(u = -2, -1, 0, 1, 2\) and \(u^2 = 4, 1, 0, 1, 4\). Then \(S_{u,u^2} = \sum u_i \cdot u_i^2 = \sum u_i^3 = -8 - 1 + 0 + 1 + 8 = 0\), so \(r = 0\) and both VIFs are \(1\). Over \(1 \le x \le 5\), \(x^2\) is an increasing function of \(x\) and nearly linear in it, hence the high correlation. After centring, \(u^2\) is symmetric about \(u = 0\) while \(u\) is antisymmetric, and for points placed symmetrically the two are exactly uncorrelated.

(c) \(\beta _0 + \beta _1u + \beta _2u^2 = (\beta _0 - 3\beta _1 + 9\beta _2) + (\beta _1 - 6\beta _2)x + \beta _2x^2\), so the two models span the same column space (the functions \(1, x, x^2\)), and least squares gives the same projection \(\hat {\mathbf {y}}\). Only the meaning, and the precision, of the individual coefficients change: in the centred model \(\beta _1\) is the slope at \(x = 3\), the middle of the data.

Solution. (a) \(Q(\mathbf {b}) = \mathbf {y}^{*\top }\mathbf {y}^* - 2\mathbf {b}^\top \mathbf {W}^\top \mathbf {y}^* + \mathbf {b}^\top (\mathbf {W}^\top \mathbf {W} + k\mathbf {I})\mathbf {b}\). By Lemma 3.1.2 the gradient is \(-2\mathbf {r}_y + 2(\mathbf {R} + k\mathbf {I})\mathbf {b}\), zero at \(\hat {\mathbf {b}}_R\). The matrix \(\mathbf {R} + k\mathbf {I}\) is positive definite, so, as in the proof of Theorem 3.3.1, \(Q(\mathbf {b}) = Q(\hat {\mathbf {b}}_R) + (\mathbf {b} - \hat {\mathbf {b}}_R)^\top (\mathbf {R} + k\mathbf {I})(\mathbf {b} - \hat {\mathbf {b}}_R)\), which is minimised only at \(\hat {\mathbf {b}}_R\).

(b) The augmented regressor matrix and response are \[ \begin {bmatrix} \mathbf {W} \\ \sqrt {k}\,\mathbf {I} \end {bmatrix} , \qquad \begin {bmatrix} \mathbf {y}^* \\ \mathbf {0} \end {bmatrix} , \] and the least-squares criterion is \(\|\mathbf {y}^* - \mathbf {Wb}\|^2 + \|\mathbf {0} - \sqrt {k}\,\mathbf {b}\|^2 = Q(\mathbf {b})\). Equivalently its normal equations are \((\mathbf {W}^\top \mathbf {W} + k\mathbf {I})\mathbf {b} = \mathbf {W}^\top \mathbf {y}^*\). The pseudo-observations say “each coefficient is probably near zero”, and they pull the estimates towards zero most strongly in the directions the real data leave undetermined.

Solution. (a) \(\mathrm {VIF}_j = 1/(1 - R_j^2)\): \(1.43\), \(20\), \(25\), \(2\). The coefficients of \(x_2\) and \(x_3\) are badly affected; those of \(x_1\) and \(x_4\) are not.

(b) The standard error is multiplied by \(\sqrt {\mathrm {VIF}_3} = 5\).

(c) Since only \(x_2\) and \(x_3\) have large VIFs, the likeliest explanation is a single near dependence between them, possibly involving the others weakly. The correlation \(r_{23}\) and the eigenvalues and eigenvectors of \(\mathbf {R}\) would confirm this: one small eigenvalue, with eigenvector concentrated on \(x_2\) and \(x_3\). Then ask whether the two measure the same thing (combine them), whether one can be dropped, or whether the aim is prediction, in which case the collinearity may not matter.

Solution. \(\sum 1/\lambda _j = 1/1.9856 + 1/0.0144 = 0.504 + 69.6 = 70.1\), so \(E\|\hat {\mathbf {b}}\|^2 - \|\mathbf {b}\|^2 = 0.01872 \times 70.1 = 1.31\). The observed \(\|\hat {\mathbf {b}}\|^2 = 1.174\) is smaller than this expected excess alone, so much of its length may well be noise. With uncorrelated regressors \(\mathbf {R} = \mathbf {I}\), both eigenvalues are \(1\), and the excess would be \(0.01872 \times 2 = 0.037\): thirty-five times smaller, the average of the two VIFs.

Solution. By the derivative in the proof of Theorem 8.4.2(iv), each term has derivative \(2\lambda _j(ka^2 - \sigma _*^2)/(\lambda _j + k)^3\), which is negative for \(k < \sigma _*^2/a^2\) and positive for \(k > \sigma _*^2/a^2\), whatever \(\lambda _j\). So every term, and hence the sum, is minimised at \(k = \sigma _*^2/a^2\). Since \(\|\mathbf {b}\|^2 = \|\boldsymbol \alpha \|^2 = qa^2\), this is \(k = q\sigma _*^2/\|\mathbf {b}\|^2\). Replacing \(\sigma _*^2\) and \(\mathbf {b}\) by their least-squares estimates gives \(\hat k = q\hat \sigma _*^2/\hat {\mathbf {b}}^\top \hat {\mathbf {b}}\).