7.2 Criteria for comparing models
Each candidate model is summarised by a number, and the numbers are compared. Write \(SS_{\mathrm {Res}}(p)\), \(R^2_p\) and \(MS_{\mathrm {Res}}(p)\) for the quantities of a subset model with \(p\) parameters, and \(\hat \sigma ^2\) for the residual mean square of the full model.
\(R^2\), adjusted \(R^2\) and the residual mean square
\(R^2_p = 1 - SS_{\mathrm {Res}}(p)/SS_T\) never decreases when a regressor is added (Theorem 3.9.1), so it cannot choose between models of different sizes; it is used to compare models of the same size, and to see where adding regressors stops paying, the point at which \(R^2_p\) levels off as \(p\) grows. The adjusted \[ R^2_{\mathrm {adj},p} = 1 - \frac {SS_{\mathrm {Res}}(p)/(n - p)}{SS_T/(n - 1)} = 1 - \frac {MS_{\mathrm {Res}}(p)}{SS_T/(n - 1)} \] does penalise size, and since \(SS_T/(n - 1)\) is the same for every model, maximising \(R^2_{\mathrm {adj},p}\) is the same as minimising \(MS_{\mathrm {Res}}(p)\). As shown in the practice problems of Section 3, adding one regressor raises \(R^2_{\mathrm {adj}}\) exactly when its partial \(F\) statistic exceeds \(1\), which is a weak requirement; these criteria tend to choose models that are too large.
Mallows’ \(C_p\)
A model is used to estimate the mean response at the observed points. A natural measure of its quality is the total mean squared error of the fitted values, scaled by \(\sigma ^2\): \[ \Gamma _p = \frac {1}{\sigma ^2}\sum _{i=1}^n E\bigl [(\hat y_i - E(y_i))^2\bigr ] . \] It combines variance and bias, and it is small for a good model.
Theorem 7.2.1. For a subset model with \(p\) parameters, \[ \Gamma _p = p + \frac {SS_B}{\sigma ^2} = \frac {E\bigl (SS_{\mathrm {Res}}(p)\bigr )}{\sigma ^2} - n + 2p , \] where \(SS_B = \sum _i\bigl (E(\hat y_i) - E(y_i)\bigr )^2 = \boldsymbol \mu ^\top (\mathbf {I} - \mathbf {H}_1)\boldsymbol \mu \) is the sum of squared biases.
Proof. Each term splits into variance plus squared bias: \(E[(\hat y_i - \mu _i)^2] = \operatorname {Var}(\hat y_i) + (E\hat y_i - \mu _i)^2\). The fitted values are \(\hat {\mathbf {y}} = \mathbf {H}_1\mathbf {y}\), with covariance matrix \(\sigma ^2\mathbf {H}_1\mathbf {H}_1^\top = \sigma ^2\mathbf {H}_1\), so \(\sum _i\operatorname {Var}(\hat y_i) = \sigma ^2\operatorname {tr}(\mathbf {H}_1) = p\sigma ^2\). The biases form the vector \(\mathbf {H}_1\boldsymbol \mu - \boldsymbol \mu = -(\mathbf {I} - \mathbf {H}_1)\boldsymbol \mu \), whose squared length is \(\boldsymbol \mu ^\top (\mathbf {I} - \mathbf {H}_1)\boldsymbol \mu = SS_B\). This gives the first form. By the proof of Theorem 7.1.2(iii), \(E(SS_{\mathrm {Res}}(p)) = (n - p)\sigma ^2 + SS_B\), so \(SS_B/\sigma ^2 = E(SS_{\mathrm {Res}}(p))/\sigma ^2 - n + p\); add \(p\). □
Definition 7.2.2. Mallows’ \(C_p\) is the estimate of \(\Gamma _p\) obtained by replacing \(E(SS_{\mathrm {Res}}(p))\) by \(SS_{\mathrm {Res}}(p)\) and \(\sigma ^2\) by the full-model estimate \(\hat \sigma ^2\): \[ C_p = \frac {SS_{\mathrm {Res}}(p)}{\hat \sigma ^2} - n + 2p . \]
If a subset model has no bias, \(SS_B = 0\) and \(\Gamma _p = p\), so its \(C_p\) should be close to \(p\). A model with substantial bias has \(C_p\) well above \(p\). The usual practice is to plot \(C_p\) against \(p\) for every candidate, draw the line \(C_p = p\), and look for models that are small, have \(C_p\) near or below the line, and have a small \(C_p\). For the full model \(C_p = p\) exactly (Problem 7.6.1), so the criterion is only as good as the assumption that the full model is unbiased and \(\hat \sigma ^2\) a fair estimate of \(\sigma ^2\).
AIC and BIC
Information criteria balance fit against size by penalising the log-likelihood. For the normal linear model with \(p\) parameters, the argument in the proof of Theorem 6.3.2 shows that the maximised log-likelihood is \(-\frac {n}{2}\ln \bigl (SS_{\mathrm {Res}}(p)/n\bigr )\) plus a constant. The Akaike information criterion and the Bayesian information criterion are \[ \mathrm {AIC}_p = n\ln \frac {SS_{\mathrm {Res}}(p)}{n} + 2p , \qquad \mathrm {BIC}_p = n\ln \frac {SS_{\mathrm {Res}}(p)}{n} + p\ln n , \] and smaller is better. The first term rewards fit; the second penalises size. Akaike derived the penalty \(2p\) as the bias in the maximised log-likelihood as an estimate of how well the model will predict new data; Schwarz derived \(p\ln n\) from a large-sample approximation to the posterior probability of the model. The derivations are beyond this course, but the consequence is easy to see: for \(n \ge 8\), \(\ln n > 2\), and BIC penalises extra parameters more heavily than AIC, so it tends to choose smaller models. As \(n \to \infty \), BIC chooses the true model (if it is among the candidates) with probability tending to \(1\), while AIC aims at good prediction and may keep a few superfluous terms.
PRESS
The prediction error sum of squares of Section 4, \(\mathrm {PRESS}_p = \sum e_i^2/(1 - h_{ii})^2\), measures how well a model predicts each observation from the others. It is a direct estimate of predictive performance, needs no estimate of \(\sigma ^2\), and is computed from a single fit of each candidate. Prefer models with small PRESS, or equivalently large \(R^2_{\mathrm {pred}}\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.