4.5 The PRESS statistic
\(R^2\) and \(MS_{\mathrm {Res}}\) measure how well a model fits the data used to fit it. A model always fits its own data better than it will fit new data, because the estimates have adapted to the particular errors in the sample. A fairer measure predicts each observation from a fit that did not use it.
Definition 4.5.1. The prediction error sum of squares is \[ \mathrm {PRESS} = \sum _{i=1}^n \bigl (y_i - \hat y_{(i)}\bigr )^2 = \sum _{i=1}^n \left (\frac {e_i}{1 - h_{ii}}\right )^2 , \] and the \(R^2\) for prediction is \[ R^2_{\mathrm {pred}} = 1 - \frac {\mathrm {PRESS}}{SS_T} . \]
The second form of PRESS is Theorem 4.2.3(iii); it means that PRESS, although it is defined by \(n\) separate fits, is computed from one. Since \(0 < 1 - h_{ii} \le 1\), every PRESS residual is at least as large as the ordinary one, so \(\mathrm {PRESS} \ge SS_{\mathrm {Res}}\) and \(R^2_{\mathrm {pred}} \le R^2\). A large gap between them warns that the model will predict new observations much less well than its fit suggests. PRESS is also useful for comparing models: other things equal, prefer the model with the smaller PRESS. Points with high leverage contribute most, because their residuals are divided by a small \(1 - h_{ii}\).
Solution. The PRESS residuals were found in Example 4.2.6. Exactly, from the leverages \(53/126\), \(37/63\), \(25/63\), \(29/126\), \[ e_{(1)} = -\frac {126}{73}, \quad e_{(2)} = \frac {63}{26}, \quad e_{(3)} = \frac {63}{38}, \quad e_{(4)} = -\frac {126}{97}, \] and \(e_{(5)} = e_{(6)} = 0\). So \[ \mathrm {PRESS} = 2.979 + 5.871 + 2.749 + 1.687 = 13.29 . \] From Example 3.8.2, \(SS_T = 93.5\), so \[ R^2_{\mathrm {pred}} = 1 - \frac {13.29}{93.5} = 0.858 , \] against \(R^2 = 0.957\). We may expect the model to explain about \(86\%\) of the variability in predicting new plots. PRESS is more than three times \(SS_{\mathrm {Res}} = 4\), a reminder that with six observations and three parameters the fit has adapted a good deal to its data. Plot 2 alone contributes \(44\%\) of PRESS: its leverage is high, and the other five plots predict it badly.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.