4.9 Variable Selection
Section 4.4 showed that \(R^2\) can never decrease when a variable is added, which makes it useless for choosing between models of different size. Section 4.5 showed that keeping variables that carry the same information produces unstable coefficients. Between them these two facts create the selection problem: which subset of the available regressors should the final model contain?
Definition 4.9.1 (Criteria for comparing subsets). For a model with \(p\) parameters fitted to \(n\) observations:
- Adjusted coefficient of determination \(R^2_a=1-\dfrac {SSE/(n-p)}{SST/(n-1)}\), which is maximised by the better model;
- Mallows’ statistic \(C_p=\dfrac {SSE_p}{s^{2}_{\text {full}}}-\left (n-2p\right )\), where \(s^2_{\text {full}}\) is the mean square error of the full model;
- Akaike’s criterion \(AIC=n\ln \left (SSE_p/n\right )+2p\) and the Bayesian criterion \(BIC=n\ln \left (SSE_p/n\right )+p\ln n\), both minimised by the better model.
Note. All four penalise size, and they differ only in how heavily. \(C_p\) has a useful reading: if the model contains all the variables that matter, then \(E\left (C_p\right )\approx p\), so subsets are plotted against \(p\) and those falling near the line \(C_p=p\) are the candidates. \(BIC\) penalises more heavily than \(AIC\) whenever \(\ln n>2\), that is for \(n\geq 8\), so it selects smaller models; and \(R^2_a\) penalises least of the four, so it tends to keep variables the others drop.
Definition 4.9.2 (Stepwise procedures).
- Forward selection starts from the model with only an intercept and at each step adds the variable with the largest partial \(F\), stopping when no candidate exceeds a chosen entry threshold.
- Backward elimination starts from the full model and at each step removes the variable with the smallest partial \(F\), stopping when every survivor exceeds a chosen removal threshold.
- Stepwise regression is forward selection which, after each addition, reconsiders every variable already present for removal.
Note. Stepwise regression reconsiders because forward selection alone cannot undo its own mistakes: a variable admitted early may be made redundant by a later pair, and without a removal step it stays in the model for good.
Note (The hazard). The partial \(F\) at each step is compared with an \(F\) table, but the variable being tested was chosen because it had the largest such statistic. Its distribution under the null is therefore not the \(F\) distribution, and the \(p\)-values reported by a stepwise procedure are too small — often by a great deal. With enough candidate regressors and pure noise for a response, a stepwise search will return a model of apparently significant variables and a respectable \(R^2\).
Three consequences follow. Selection is exploratory, and the model it returns is a hypothesis rather than a result. The \(F\) and \(t\) statistics printed for the selected model do not have their nominal distributions and should not be quoted as though they do. And a model chosen this way should be assessed on data not used to choose it — by splitting the sample, or by cross-validation — because that is the only assessment the selection cannot corrupt.
Note. Selection is also not a remedy for the multicollinearity of Section 4.5, though it is often used as one. Dropping one of two collinear regressors does stabilise the coefficients, but which of the two gets dropped is close to arbitrary — decided by noise — and the surviving coefficient must then be read as carrying the effect of both. If the aim is prediction this is harmless; if the aim is to interpret an individual coefficient it is not.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.