Solution. (a) For the full model \(SS_{\mathrm {Res}} = (n - p)\hat \sigma ^2\), so \(C_p = (n - p) - n + 2p = p\).

(b) \(C_p < 0\) when \(SS_{\mathrm {Res}}(p)/\hat \sigma ^2 < n - 2p\), that is, \(MS_{\mathrm {Res}}(p) < \hat \sigma ^2(n - 2p)/(n - p)\), which is possible since \(MS_{\mathrm {Res}}(p)\) is a random variable that can fall below \(\hat \sigma ^2\). Since \(\Gamma _p \ge p > 0\), a negative \(C_p\) is pure sampling error, and usually means that the subset model is unbiased and its residual mean square happened to be small; it can also mean that \(\hat \sigma ^2\) is inflated by superfluous terms in the full model.

(c) \(R^2_{\mathrm {adj},p} = 1 - MS_{\mathrm {Res}}(p)\big /[SS_T/(n - 1)]\), and \(SS_T/(n - 1)\) is the same for every model fitted to the same responses. So \(R^2_{\mathrm {adj}}\) is a decreasing function of \(MS_{\mathrm {Res}}(p)\).

Solution. (a) By Theorem 7.1.2(i), \(E(\hat {\boldsymbol \beta }_1^*) = \boldsymbol \beta _1 + \mathbf {A}\beta _2\), where \(\mathbf {A}\) holds the coefficients of the simple regression of \(x_2\) on \(x_1\). Its slope is \(S_{12}/S_{11}\). (Directly: \(\hat \beta _1^* = \sum (x_{i1} - \bar x_1)y_i/S_{11}\), and substituting \(E(y_i) = \beta _0 + \beta _1x_{i1} + \beta _2x_{i2}\) gives \(\beta _1 + \beta _2S_{12}/S_{11}\), since \(\sum (x_{i1} - \bar x_1) = 0\).)

(b) When \(\beta _2 = 0\), or when \(S_{12} = 0\), so that \(x_1\) and \(x_2\) are uncorrelated in the sample. This is one reason for designing experiments with uncorrelated (orthogonal) factors.

(c) An omitted variable, such as household income, raises both the number of radios and school attendance. In the regression of attendance on radios alone, the radio coefficient is \(\beta _1 + \beta _2S_{12}/S_{11}\), where \(\beta _2\) is the effect of income and \(S_{12}/S_{11}\) the slope of income on radios. Even if radios have no effect (\(\beta _1 = 0\)), the estimate is positive.

Solution. (a) \(\mathrm {AIC}_{p+1} < \mathrm {AIC}_p\) means \(n\ln SS_{\mathrm {Res}}(p + 1) + 2(p + 1) < n\ln SS_{\mathrm {Res}}(p) + 2p\) (the \(-n\ln n\) terms cancel), that is, \(\ln [SS_{\mathrm {Res}}(p)/SS_{\mathrm {Res}}(p + 1)] > 2/n\). For BIC the penalty difference is \(\ln n\) instead of \(2\), so the condition is \(SS_{\mathrm {Res}}(p)/SS_{\mathrm {Res}}(p + 1) > n^{1/n}\).

(b) The partial \(F\) statistic for the added term is \(F = (n - p - 1)\bigl [SS_{\mathrm {Res}}(p)/SS_{\mathrm {Res}}(p + 1) - 1\bigr ]\). With \(n = 12\) and, say, \(p = 2\) (so \(n - p - 1 = 9\)): \begin {align*} \text {AIC:}\quad F &> 9\,(e^{2/12} - 1) = 9 \times 0.1814 = 1.63, \\ \text {BIC:}\quad F &> 9\,(12^{1/12} - 1) = 9 \times 0.2300 = 2.07 . \end {align*}

Both are more lenient than \(F_{\mathrm {in}} = 4\), and AIC more than BIC. With small samples, AIC in particular tends to admit terms that a significance test would reject.

Solution. (a) Forward. Step 1: the single-regressor \(F\) values are \((474.5 - 39.97)/(39.97/8) = 87.0\), \((474.5 - 82.66)/(82.66/8) = 37.9\) and \((474.5 - 178.09)/(178.09/8) = 13.3\); \(x_1\) enters. Step 2: given \(x_1\), adding \(x_2\) gives \(F = (39.97 - 32.56)/(32.56/7) = 1.59\), and adding \(x_3\) gives \((39.97 - 33.59)/(33.59/7) = 1.33\). Both are below \(4\), so forward selection stops with \(x_1\) alone, \(SS_{\mathrm {Res}} = 39.97\).

Backward. In the full model \(MS_{\mathrm {Res}} = 3.40/6 = 0.566\). Deleting \(x_1\) gives \(F = (3.42 - 3.40)/0.566 = 0.05\); deleting \(x_2\), \((33.59 - 3.40)/0.566 = 53.3\); deleting \(x_3\), \((32.56 - 3.40)/0.566 = 51.5\). Remove \(x_1\). In the model \(x_2, x_3\) (\(MS_{\mathrm {Res}} = 3.42/7 = 0.489\)), deleting \(x_2\) gives \(F = (178.09 - 3.42)/0.489 = 357\) and deleting \(x_3\) gives \((82.66 - 3.42)/0.489 = 162\). Stop with \(x_2, x_3\), \(SS_{\mathrm {Res}} = 3.42\).

(b) With \(\hat \sigma ^2 = 0.566\) and \(n = 10\): \(C_p(x_1) = 39.97/0.566 - 10 + 4 = 64.6\), \(C_p(x_2x_3) = 3.42/0.566 - 10 + 6 = 2.04\), and every other model with \(p < 4\) has \(C_p > 50\). BIC is likewise smallest for \(x_2, x_3\) (\(10\ln 0.342 + 3\ln 10 = -3.81\), against \(18.5\) for \(x_1\)). Both choose \(x_2, x_3\).

(c) Because \(x_1\) is close to the average of \(x_2\) and \(x_3\), it is the best single stand-in for both, and forward selection, which looks one regressor at a time, takes it first. Once \(x_1\) is in, \(x_2\) and \(x_3\) each add little separately, since \(x_1\) already carries most of what each contributes, so neither enters. But \(y\) depends on \(x_2\) and \(x_3\) with different coefficients, which \(x_1\) cannot mimic, and the pair together fits ten times better. Backward elimination starts from the model that contains the pair and finds that \(x_1\) is the redundant one. When regressors are correlated, one-at-a-time searches can miss the best subset; this is the strongest argument for examining all subsets when it is feasible.

Solution. \(R^2_{\mathrm {pred}} = 1 - \mathrm {PRESS}/SS_T\) with \(SS_T = 574.67\): for \(x_1, x_2\), \(1 - 46.28/574.67 = 0.919\); for the full model, \(1 - 68.74/574.67 = 0.880\). The full model fits the data slightly better but predicts new farms worse. The extra term has adapted to noise in these twelve farms, and that noise does not recur. The gap between \(R^2\) and \(R^2_{\mathrm {pred}}\) is \(0.032\) for the smaller model and \(0.073\) for the larger, a typical symptom of overfitting.