6.3 The decomposition of the sum of squares and \(R^2\)
Theorem 6.3.1. If the model contains an intercept, then with \(M_0\) the centring matrix, \[ \underbrace {Y'M_0Y}_{\text {TSS}} = \underbrace {\hat Y'M_0\hat Y}_{\text {ESS}} + \underbrace {e'e}_{\text {RSS}}, \qquad \text {ESS} = b'X'Y - n\bar Y^2 . \]
Proof. \(Y = \hat Y + e\), so \(M_0Y = M_0\hat Y + M_0e = M_0\hat Y + e\), because \(\sum e_i = 0\) gives \(M_0e = e\). Then \[ Y'M_0Y = (M_0Y)'(M_0Y) = \hat Y'M_0\hat Y + 2\hat Y'M_0e + e'e , \] and \(\hat Y'M_0e = \hat Y'e = 0\) by Theorem 6.2.2. For the second formula, \(\hat Y'M_0\hat Y = \hat Y'\hat Y - n\bar Y^2\) since \(\bar {\hat Y} = \bar Y\), and \(\hat Y'\hat Y = b'X'Xb = b'X'Y\) by the normal equations. □
As before, \(R^2 = \text {ESS}/\text {TSS} = 1 - \text {RSS}/\text {TSS}\). Without an intercept, \(M_0e \neq e\) in general, the decomposition fails, and \(1 - \text {RSS}/\text {TSS}\) can be negative (compare Problem 2.7.4).
Theorem 6.3.2. Adding a regressor to a model never reduces \(R^2\): if the regressors of model 1 are a subset of those of model 2, both with an intercept and the same dependent variable, then \(R_2^2 \geq R_1^2\).
Proof. Model 2 minimises the residual sum of squares over a set of coefficient vectors that includes every fit available to model 1, namely those with the extra coefficients set to zero. Its minimum can be no larger: \(\text {RSS}_2 \leq \text {RSS}_1\). Both have the same TSS, so \(R_2^2 = 1 - \text {RSS}_2/\text {TSS} \geq R_1^2\). □
So a reported \(R^2_{1\cdot 23} = 0.87\) together with \(R^2_{1\cdot 234} = 0.82\), for the same data, is impossible, whatever the variables. The other side of the coin is that \(R^2\) rewards adding variables whether or not they matter: even a column of random numbers raises it (Software 6.7.3). The adjusted \(R^2\) corrects each sum of squares for its degrees of freedom, \[ \bar R^2 = 1 - \frac {\text {RSS}/(n - k)}{\text {TSS}/(n - 1)} = 1 - (1 - R^2)\frac {n - 1}{n - k} , \] and can fall when a regressor is added. It rises precisely when the added variable’s \(t\) ratio exceeds one in absolute value (Problem 6.8.2). \(\bar R^2\) can be negative, and it is a crude device. Its proper successors are the formal tests and information criteria introduced later.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.