Proof. The normal equations of the full model, split into the rows belonging to \(\mathbf {X}_1\) and to \(\mathbf {X}_2\), are \begin {align*} \mathbf {X}_1^\top \mathbf {X}_1\hat {\boldsymbol \beta }_1 + \mathbf {X}_1^\top \mathbf {X}_2\hat {\boldsymbol \beta }_2 &= \mathbf {X}_1^\top \mathbf {y} , \\ \mathbf {X}_2^\top \mathbf {X}_1\hat {\boldsymbol \beta }_1 + \mathbf {X}_2^\top \mathbf {X}_2\hat {\boldsymbol \beta }_2 &= \mathbf {X}_2^\top \mathbf {y} . \end {align*}

(i) Multiply the first by \((\mathbf {X}_1^\top \mathbf {X}_1)^{-1}\): \(\hat {\boldsymbol \beta }_1 + \mathbf {A}\hat {\boldsymbol \beta }_2 = \hat {\boldsymbol \beta }_1^*\).

(ii) Substitute \(\hat {\boldsymbol \beta }_1 = \hat {\boldsymbol \beta }_1^* - \mathbf {A}\hat {\boldsymbol \beta }_2\) in the second. Since \(\mathbf {X}_1\hat {\boldsymbol \beta }_1^* = \mathbf {H}_1\mathbf {y}\) and \(\mathbf {X}_1\mathbf {A} = \mathbf {H}_1\mathbf {X}_2\), \[ \mathbf {X}_2^\top \mathbf {H}_1\mathbf {y} - \mathbf {X}_2^\top \mathbf {H}_1\mathbf {X}_2\hat {\boldsymbol \beta }_2 + \mathbf {X}_2^\top \mathbf {X}_2\hat {\boldsymbol \beta }_2 = \mathbf {X}_2^\top \mathbf {y} , \] that is, \(\mathbf {X}_2^\top (\mathbf {I} - \mathbf {H}_1)\mathbf {X}_2\,\hat {\boldsymbol \beta }_2 = \mathbf {X}_2^\top (\mathbf {I} - \mathbf {H}_1)\mathbf {y}\). The matrix on the left is invertible because \(\mathbf {X}\) has full column rank (the argument is that of Theorem 3.3.1, applied to \((\mathbf {I} - \mathbf {H}_1)\mathbf {X}_2\)).

(iii) By Lemma 3.1.3, the covariance of \((\mathbf {X}_1^\top \mathbf {X}_1)^{-1}\mathbf {X}_1^\top \mathbf {y}\) with \(\mathbf {M}\mathbf {X}_2^\top (\mathbf {I} - \mathbf {H}_1)\mathbf {y}\), where \(\mathbf {M}\) is the inverse in (ii), is \(\sigma ^2(\mathbf {X}_1^\top \mathbf {X}_1)^{-1}\mathbf {X}_1^\top (\mathbf {I} - \mathbf {H}_1)\mathbf {X}_2\mathbf {M} = \mathbf {0}\), because \(\mathbf {X}_1^\top (\mathbf {I} - \mathbf {H}_1) = \bigl ((\mathbf {I} - \mathbf {H}_1)\mathbf {X}_1\bigr )^\top = \mathbf {0}\). □

Proof. (i) \(E(\mathbf {y}) = \mathbf {X}_1\boldsymbol \beta _1 + \mathbf {X}_2\boldsymbol \beta _2\), so \(E(\hat {\boldsymbol \beta }_1^*) = (\mathbf {X}_1^\top \mathbf {X}_1)^{-1}\mathbf {X}_1^\top (\mathbf {X}_1\boldsymbol \beta _1 + \mathbf {X}_2\boldsymbol \beta _2) = \boldsymbol \beta _1 + \mathbf {A}\boldsymbol \beta _2\).

(ii) By Lemma 7.1.1(i), \(\hat {\boldsymbol \beta }_1 = \hat {\boldsymbol \beta }_1^* - \mathbf {A}\hat {\boldsymbol \beta }_2\), a difference of uncorrelated vectors by part (iii) of the lemma, so the variances add. The diagonal entries of \(\mathbf {A}\operatorname {Var}(\hat {\boldsymbol \beta }_2)\mathbf {A}^\top \) are variances of linear combinations of \(\hat {\boldsymbol \beta }_2\), and so are \(\ge 0\).

(iii) Write \(\boldsymbol \mu = \mathbf {X}_1\boldsymbol \beta _1 + \mathbf {X}_2\boldsymbol \beta _2\). \(SS_{\mathrm {Res}}(p) = \mathbf {y}^\top (\mathbf {I} - \mathbf {H}_1)\mathbf {y}\), and by Lemma 3.1.4 \begin {align*} E\bigl (SS_{\mathrm {Res}}(p)\bigr ) &= \sigma ^2\operatorname {tr}(\mathbf {I} - \mathbf {H}_1) + \boldsymbol \mu ^\top (\mathbf {I} - \mathbf {H}_1)\boldsymbol \mu \\ &= (n - p)\sigma ^2 + \boldsymbol \beta _2^\top \mathbf {X}_2^\top (\mathbf {I} - \mathbf {H}_1)\mathbf {X}_2\boldsymbol \beta _2 , \end {align*}

since \((\mathbf {I} - \mathbf {H}_1)\mathbf {X}_1 = \mathbf {0}\). The quadratic form is \(\|(\mathbf {I} - \mathbf {H}_1)\mathbf {X}_2\boldsymbol \beta _2\|^2 \ge 0\).

(iv) The mean square error matrix is the variance plus the bias times its transpose: \(\operatorname {Var}(\hat {\boldsymbol \beta }_1^*) + \mathbf {A}\boldsymbol \beta _2\boldsymbol \beta _2^\top \mathbf {A}^\top \). Substitute \(\operatorname {Var}(\hat {\boldsymbol \beta }_1^*) = \operatorname {Var}(\hat {\boldsymbol \beta }_1) - \mathbf {A}\operatorname {Var}(\hat {\boldsymbol \beta }_2)\mathbf {A}^\top \) from (ii). □

Solution. By the methods of Section 3 (with the matrices given at the end of this solution):

Underfitting. Leaving out labour raises the fertiliser coefficient from \(1.98\) to \(2.75\). The alias matrix explains it: regressing \(x_2\) on \(x_1\) gives slope \(38.5/65 = 0.592\), and by Lemma 7.1.1(i), \[ \hat \beta _1^* = \hat \beta _1 + 0.592\,\hat \beta _2 = 1.982 + 0.592 \times 1.303 = 2.754 . \] Farms that use more fertiliser also tend to put in more labour, and with labour omitted, fertiliser is credited with part of its effect. The residual mean square rises from \(3.15\) to \(8.17\).

Overfitting. Adding the distance to market, whose coefficient is small and far from significant (\(t = 0.65\)), raises the standard error of \(\hat \beta _1\) from \(0.289\) to \(0.407\) and of \(\hat \beta _2\) from \(0.316\) to \(0.424\): distance is mildly correlated with the other two, and carrying it along costs precision for nothing.

For the record, for the model in \(x_1, x_2\), \[ \mathbf {X}^\top \mathbf {X} = \begin {bmatrix} 12 & 42 & 45 \\ 42 & 212 & 196 \\ 45 & 196 & 223 \end {bmatrix} , \qquad \mathbf {X}^\top \mathbf {y} = \begin {bmatrix} 212 \\ 921 \\ 942 \end {bmatrix} , \] with \(\det (\mathbf {X}^\top \mathbf {X}) = 24528\), and the fit is \(\hat y = 5.843 + 1.982x_1 + 1.303x_2\).