Proof. (i) \(\operatorname {Var}(\hat {\boldsymbol \alpha }) = \mathbf {P}^\top (\sigma _*^2\mathbf {R}^{-1})\mathbf {P} = \sigma _*^2\mathbf {P}^\top \mathbf {P}\boldsymbol \Lambda ^{-1}\mathbf {P}^\top \mathbf {P} = \sigma _*^2\boldsymbol \Lambda ^{-1}\), a diagonal matrix; and \(E(\hat {\boldsymbol \alpha }) = \mathbf {P}^\top \mathbf {b} = \boldsymbol \alpha \).

(ii) By the normal equations \(\mathbf {r}_y = \mathbf {R}\hat {\mathbf {b}}\), so \(\hat {\mathbf {b}}_R = (\mathbf {R} + k\mathbf {I})^{-1}\mathbf {R}\hat {\mathbf {b}} = \mathbf {P}(\boldsymbol \Lambda + k\mathbf {I})^{-1}\boldsymbol \Lambda \mathbf {P}^\top \hat {\mathbf {b}}\), using \(\mathbf {R} + k\mathbf {I} = \mathbf {P}(\boldsymbol \Lambda + k\mathbf {I})\mathbf {P}^\top \). Multiply on the left by \(\mathbf {P}^\top \).

(iii) \(\mathbf {P}\) is orthogonal, so \(\|\hat {\mathbf {b}}_R - \mathbf {b}\|^2 = \|\hat {\boldsymbol \alpha }_R - \boldsymbol \alpha \|^2\), and each component’s mean squared error is its variance plus its squared bias. By (i) and (ii), with \(c_j = \lambda _j/(\lambda _j + k)\), the variance is \(c_j^2\sigma _*^2/\lambda _j = \sigma _*^2\lambda _j/(\lambda _j + k)^2\) and the bias is \((c_j - 1)\alpha _j = -k\alpha _j/(\lambda _j + k)\).

(iv) Differentiate one term of (iii) with respect to \(k\): \begin {align*} \frac {d}{dk}\,\frac {\sigma _*^2\lambda _j + k^2\alpha _j^2}{(\lambda _j + k)^2} &= \frac {2k\alpha _j^2(\lambda _j + k) - 2(\sigma _*^2\lambda _j + k^2\alpha _j^2)}{(\lambda _j + k)^3} \\ &= \frac {2\lambda _j(k\alpha _j^2 - \sigma _*^2)}{(\lambda _j + k)^3} . \end {align*}

If \(k < \sigma _*^2/\alpha _j^2\) for every \(j\), each derivative is negative, so \(\mathrm {MSE}(k)\) is strictly decreasing on that interval, which starts at \(k = 0\). □

Solution. In correlation form, \(r_{12} = 0.98563\), \(r_{1y} = 0.95182\) and \(r_{2y} = 0.93455\), and the least-squares standardized coefficients are \(\hat b_1 = 1.0760\) and \(\hat b_2 = -0.1260\). (Check: \(\hat \beta _1 = 1.0760\sqrt {163.5/65.5} = 1.70\).) For a given \(k\), \[ \hat {\mathbf {b}}_R = \frac {1}{(1 + k)^2 - r_{12}^2}\begin {bmatrix} (1 + k)r_{1y} - r_{12}r_{2y} \\ (1 + k)r_{2y} - r_{12}r_{1y} \end {bmatrix} . \] For instance at \(k = 0.05\), \((1.05)^2 - 0.97147 = 0.13103\), and \[ \hat {\mathbf {b}}_R = \frac {1}{0.13103}\begin {bmatrix} 0.99941 - 0.92112 \\ 0.98128 - 0.93815 \end {bmatrix} = \begin {bmatrix} 0.5975 \\ 0.3292 \end {bmatrix} . \] Converting each to the original scale with \(\hat \beta _j = \hat b_j\sqrt {SS_T/S_{jj}}\):

The ridge trace, Figure 16, shows the two coefficients moving rapidly towards each other for small \(k\), the sign of \(\hat b_2\) turning positive by \(k = 0.005\), and both settling down by about \(k = 0.05\). For the Hoerl–Kennard–Baldwin value, \(\hat \sigma _*^2 = MS_{\mathrm {Res}}/SS_T = 3.06/163.5 = 0.01872\) and \(\hat {\mathbf {b}}^\top \hat {\mathbf {b}} = 1.0760^2 + 0.1260^2 = 1.1736\), so \[ \hat k = \frac {2 \times 0.01872}{1.1736} = 0.032 , \] giving \(\hat {\mathbf {b}}_R = (0.654,\ 0.281)\) and, on the original scale, with \(\hat \beta _{R,0} = \bar y - \hat \beta _{R,1}\bar x_1 - \hat \beta _{R,2}\bar x_2\), \[ \hat y_R = 4.00 + 1.03x_1 + 0.45x_2 . \] Both coefficients are now positive and of sensible size: about one unit of use per appliance, and a further half unit for each one used daily. The fit has hardly suffered: \(R^2\) falls from \(0.906\) to \(0.901\).

The theory explains the result. The eigenvalues are \(1.9856\) and \(0.0144\), and the least-squares components along the eigenvectors are \(\hat \alpha _1 = (1.0760 - 0.1260)/\sqrt 2 = 0.672\) and \(\hat \alpha _2 = (1.0760 + 0.1260)/\sqrt 2 = 0.850\), the latter with standard error \(\sqrt {0.01872/0.0144} = 1.14\), larger than the estimate itself. At \(k = 0.032\) ridge multiplies \(\hat \alpha _1\) by \(1.9856/2.0176 = 0.98\), leaving it almost unchanged, and \(\hat \alpha _2\) by \(0.0144/0.0464 = 0.31\), removing most of the noise in the poorly determined direction. If the estimates were the true values, Theorem 8.4.2(iii) would give \(\mathrm {MSE}(0) = 1.31\) and \(\mathrm {MSE}(0.032) = 0.48\), a reduction by more than half.