7.5 Cautions in model selection

(i)

There is no single best model. Different criteria, and different search methods, can choose different models, and several models are often nearly equally good. Variable selection should produce a short list of candidates, to be judged by residual analysis, by subject-matter knowledge, and by how sensible the coefficients are.

(ii)

Stepwise methods do not search every model. Forward selection never removes a regressor; backward elimination never brings one back. When regressors are correlated, a variable that looks best alone may be useless in combination, and the two methods can end in different places, neither of them the best subset (Problem 7.6.4).

(iii)

Inference after selection is optimistic. The \(t\) and \(F\) statistics of the chosen model are computed as if it had been specified in advance. It was not: it was chosen because its statistics looked good. Its \(P\)-values are too small, its \(R^2\) too large and its residual mean square too small. Where possible, a model chosen on one set of data should be checked on fresh data, or on a part of the data held back for the purpose (validation).

(iv)

Respect the structure of the model. If \(x^2\) or an interaction \(x_1x_2\) is kept, the lower-order terms \(x\), \(x_1\) and \(x_2\) are normally kept too, whatever their individual \(t\) values, so that the model does not depend on the arbitrary origin of the measurement scale.

(v)

Multicollinearity makes selection unstable. When regressors are strongly correlated, small changes in the data can change which of them is chosen. This is the subject of the next section.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.