7 Selecting the Best Regression Model

So far the regressors in a model have been given. In practice there is usually a list of candidates, some of which matter and some of which do not, and part of the analysis is to decide which to keep. This is the variable selection problem. Two forces pull against each other. Leaving out a regressor that matters biases the estimates of the others and the predictions. Putting in regressors that do not matter costs degrees of freedom and inflates the variances. A good model is a compromise: as simple as it can be while still describing the data adequately.

This section first makes the two costs precise, then describes the criteria used to compare candidate models, and finally the two ways of searching among them: examining every possible subset, and stepwise methods that add or remove one regressor at a time.

Throughout, the full model contains all \(K\) candidate regressors, and a subset model contains \(p - 1\) of them with an intercept, so \(p\) parameters. Model adequacy checking (Section 4) should be done on the full model first; selection among regressors assumes that the form of the model is otherwise right.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.