Solution. (a) With \(\hat \beta _1 = 4/7\) exactly, the fitted values are \(46.3333 + 0.5714x_i\), giving

In exact terms the residuals are \(-1/21\), \(5/21\), \(-10/21\), \(17/21\), \(-19/21\), \(8/21\); they add to zero, and \(\sum (x_i - 35)e_i = 0\). Then \(SS_{\mathrm {Res}} = 840/441 = 1.905\) and \(MS_{\mathrm {Res}} = 1.905/4 = 0.476\). The leverages are \(h_{ii} = 1/6 + (x_i - 35)^2/1750\), from Example 4.1.2.

(b) With \(r_i = e_i/\sqrt {0.476(1 - h_{ii})}\) and \(S^2_{(i)} = (1.905 - e_i^2/(1 - h_{ii}))/3\):

The largest \(|t_i|\) is \(2.165\), far below the Bonferroni critical value \(t_{0.05/12,\,3} = 6.23\). No observation is an outlier.

(c) The PRESS residuals \(e_i/(1 - h_{ii})\) are \(-0.100\), \(0.338\), \(-0.581\), \(0.988\), \(-1.284\), \(0.800\), so \[ \mathrm {PRESS} = 0.010 + 0.114 + 0.338 + 0.977 + 1.648 + 0.640 = 3.727 . \] \(SS_T = 26974 - 398^2/6 = 573.33\), so \(R^2_{\mathrm {pred}} = 1 - 3.727/573.33 = 0.9935\), very close to \(R^2 = 0.9967\). The line should predict new runs about as well as it fits these.

Solution. (a) \(\mathbf {e} = (\mathbf {I} - \mathbf {H})\mathbf {y}\) and \(\operatorname {Var}(\mathbf {y}) = \sigma ^2\mathbf {I}\), so \(\operatorname {Cov}(\mathbf {e}, \mathbf {y}) = (\mathbf {I} - \mathbf {H})\sigma ^2\mathbf {I} = \sigma ^2(\mathbf {I} - \mathbf {H})\). In particular \(\operatorname {Cov}(e_i, y_i) = \sigma ^2(1 - h_{ii}) > 0\) whenever \(h_{ii} < 1\).

(b) With an intercept, \(\sum e_i = 0\), so \(\bar e = 0\) and \[ \sum (e_i - \bar e)(y_i - \bar y) = \sum e_i y_i = \mathbf {e}^\top \mathbf {y} . \] Since \(\mathbf {y} = \hat {\mathbf {y}} + \mathbf {e}\) and \(\mathbf {e}^\top \hat {\mathbf {y}} = 0\) (Theorem 3.4.2), this is \(\mathbf {e}^\top \mathbf {e} = SS_{\mathrm {Res}}\). The sums of squares about the means are \(\sum e_i^2 = SS_{\mathrm {Res}}\) and \(SS_T\). So the correlation is \[ \frac {SS_{\mathrm {Res}}}{\sqrt {SS_{\mathrm {Res}}\, SS_T}} = \sqrt {\frac {SS_{\mathrm {Res}}}{SS_T}} = \sqrt {1 - R^2} . \]

(c) The correlation in (b) is positive for every data set, whether the model is right or not. A plot of \(e_i\) against \(y_i\) would always show a rising trend, which would be a property of least squares and not of the data. Against \(\hat y_i\) the sample correlation is exactly zero (since \(\mathbf {e}^\top \hat {\mathbf {y}} = 0\) and \(\bar e = 0\)), so any pattern seen there is genuine. For the catalyst data, \(\sqrt {1 - 0.749} = 0.50\): a plot of \(e\) against \(y\) would have shown a misleading upward drift.

Solution. The slope of the least-squares line of \(w_i = e_i + \hat \beta _j x_{ij}\) on \(x_{ij}\) is \[ \frac {\sum (x_{ij} - \bar x_j) w_i}{\sum (x_{ij} - \bar x_j)^2} = \frac {\sum (x_{ij} - \bar x_j) e_i + \hat \beta _j \sum (x_{ij} - \bar x_j) x_{ij}}{\sum (x_{ij} - \bar x_j)^2} . \] By Theorem 3.4.2(iii), \(\sum x_{ij}e_i = 0\) and \(\sum e_i = 0\), so the first sum in the numerator vanishes. The second sum, \(\sum (x_{ij} - \bar x_j)x_{ij}\), equals \(\sum (x_{ij} - \bar x_j)^2\), so the slope is \(\hat \beta _j\).

For the crop data, \(\hat \beta _1 = 2\) and the residuals are \(-1, 1, 1, -1, 0, 0\), so the partial residuals \(e_i + 2x_{i1}\) are \(1, 5, 5, 5, 8, 12\) at \(x_{i1} = 1, 2, 2, 3, 4, 6\). With \(\bar x_1 = 3\) and \(\sum (x_{i1} - 3)^2 = 16\), \[ \frac {-2(1) - 1(5) - 1(5) + 0(5) + 1(8) + 3(12)}{16} = \frac {32}{16} = 2 . \]

Solution. Apply Theorem 3.11.4 with \(\mathbf {d}_i\) as the added column and \(\mathbf {X}\) as the others. Then \(\mathbf {u} = (\mathbf {I} - \mathbf {H})\mathbf {d}_i\), column \(i\) of \(\mathbf {I} - \mathbf {H}\), and since \(\mathbf {I} - \mathbf {H}\) is symmetric and idempotent, \[ \mathbf {u}^\top \mathbf {u} = \mathbf {d}_i^\top (\mathbf {I} - \mathbf {H})\mathbf {d}_i = 1 - h_{ii}, \qquad \mathbf {u}^\top \mathbf {y} = \mathbf {d}_i^\top (\mathbf {I} - \mathbf {H})\mathbf {y} = e_i . \] (a) \(\hat \gamma = \mathbf {u}^\top \mathbf {y}/\mathbf {u}^\top \mathbf {u} = e_i/(1 - h_{ii})\).

(b) The extra sum of squares due to \(\gamma \) is \((\mathbf {u}^\top \mathbf {y})^2/\mathbf {u}^\top \mathbf {u} = e_i^2/(1 - h_{ii})\), so the enlarged model has residual sum of squares \(SS_{\mathrm {Res}} - e_i^2/(1 - h_{ii})\), on \(n - p - 1\) degrees of freedom. By Theorem 4.2.3(iv) its mean square is \(S^2_{(i)}\). (This is no coincidence: with \(\gamma \) free, observation \(i\) is fitted exactly and the other \(n - 1\) determine \(\boldsymbol \beta \).)

(c) The coefficient \(\gamma \) has \(C_{\gamma \gamma } = 1/\mathbf {u}^\top \mathbf {u} = 1/(1 - h_{ii})\), so \[ t_0 = \frac {\hat \gamma }{\sqrt {S^2_{(i)}/(1 - h_{ii})}} = \frac {e_i/(1 - h_{ii})}{\sqrt {S^2_{(i)}/(1 - h_{ii})}} = \frac {e_i}{\sqrt {S^2_{(i)}(1 - h_{ii})}} = t_i . \] By Theorem 3.10.1 it has the \(t_{n - p - 1}\) distribution when \(\gamma = 0\), which is a second proof of Theorem 4.2.5(ii).

Solution. From the proof of Theorem 3.4.2(v), \(h_{ii} = h_{ii}^2 + \sum _{j \neq i}h_{ij}^2\). If \(h_{ii} = 1\) this gives \(\sum _{j \neq i}h_{ij}^2 = 0\), so \(h_{ij} = 0\) for \(j \neq i\), and row \(i\) of \(\mathbf {H}\) is \(\mathbf {d}_i^\top \). Then \(\hat y_i = \sum _j h_{ij}y_j = y_i\), so \(e_i = 0\) whatever \(\mathbf {y}\) is.

The fitted surface is forced through observation \(i\): one parameter’s worth of the model is determined by that observation alone, so the data contain no check of it. Its residual is zero not because it fits but because it cannot fail to. The PRESS residual \(e_i/(1 - h_{ii})\) is \(0/0\), undefined, and correspondingly the fit without observation \(i\) cannot be computed. With \(\mathbf {v} = (\mathbf {X}^\top \mathbf {X})^{-1}\mathbf {x}_i \neq \mathbf {0}\), \(\mathbf {Xv}\) is column \(i\) of \(\mathbf {H}\), which is \(\mathbf {d}_i\); so every row of \(\mathbf {X}_{(i)}\) is orthogonal to \(\mathbf {v}\), \(\mathbf {X}_{(i)}\mathbf {v} = \mathbf {0}\), and \(\mathbf {X}_{(i)}\) does not have full column rank. Such a point is maximally influential, and it is studied in the next section.

An example: \(y = \beta _0 + \beta _1 x + \varepsilon \) with every observation at \(x = 0\) except one, at \(x = 1\). Then \(\beta _0\) is estimated by the mean of the others and the slope is determined by the single point at \(x = 1\), which the line must pass through. Indeed \(\bar x = 1/n\), \(S_{xx} = 1 - 1/n\), and \(h = 1/n + (1 - 1/n)^2/(1 - 1/n) = 1/n + 1 - 1/n = 1\).

Solution. (a) \(n = 9\), \(\sum x_i = 25\), \(\sum y_i = 68\), \(\sum x_i^2 = 91\), \(\sum x_i y_i = 232\), \(\sum y_i^2 = 606\). Then \begin {align*} S_{xx} &= 91 - \frac {625}{9} = \frac {194}{9}, \\ S_{xy} &= 232 - \frac {25 \times 68}{9} = \frac {388}{9}, \end {align*}

so \(\hat \beta _1 = 388/194 = 2\) and \(\hat \beta _0 = 68/9 - 2 \times 25/9 = 18/9 = 2\): \[ \hat y = 2 + 2x . \] (b) The fitted values at \(x = 1, \dots , 5\) are \(4, 6, 8, 10, 12\), and the residuals are \(-1\), \(0\), \(1\), \(-1\), \(0\), \(1\), \(1\), \(-1\), \(0\), so \(SS_{\mathrm {Res}} = 6\) on \(7\) degrees of freedom. There are \(m = 5\) distinct values of \(x\). Pure error comes from the replicated levels: at \(x = 1\), mean \(4\), \(SS = 1 + 0 + 1 = 2\); at \(x = 3\), mean \(8.5\), \(SS = 0.5\); at \(x = 5\), mean \(11.5\), \(SS = 0.5\). So \(SS_{\mathrm {PE}} = 3\) on \(n - m = 4\) degrees of freedom, and \(SS_{\mathrm {LOF}} = 6 - 3 = 3\) on \(m - 2 = 3\). Also \[ SS_T = 606 - \frac {68^2}{9} = 92.22, \qquad SS_R = \hat \beta _1 S_{xy} = \frac {2 \times 388}{9} = 86.22 , \] and indeed \(86.22 + 6 = 92.22\).

\(F_0 = 1.00/0.75 = 1.33 < F_{0.05,3,4} = 6.59\) (\(P = 0.38\)). There is no evidence of lack of fit.

(c) With no evidence against the line, the pooled estimate is used: \(\hat \sigma ^2 = MS_{\mathrm {Res}} = 6/7 = 0.857\), on \(7\) degrees of freedom rather than the \(4\) of \(MS_{\mathrm {PE}}\).

Solution. Here \(h = 0.1\), \(SS_{\mathrm {Res}} = 15.1\), \(n - p = 8\). By Theorem 4.2.3(iv), \[ S^2_{(6)} = \frac {15.1 - 2.3^2/0.9}{7} = \frac {15.1 - 5.878}{7} = 1.317 , \] \[ t_6 = \frac {2.3}{\sqrt {1.317 \times 0.9}} = \frac {2.3}{1.089} = 2.11 . \] The Bonferroni critical value is \(t_{0.05/20,\,7} = 4.03\), so the observation is not an outlier. That is the right conclusion: no single observation is out of line. The two largest positive residuals both sit at \(x = 3\), and the two runs there agree closely with each other. The fault is in the model, not in any observation, and it is the lack-of-fit test, not the outlier test, that detects it. A residual analysis should always look at the pattern of the residuals as a whole, not only at the largest one.