2.9 Analysis of variance
A second way to test significance of regression splits the total variation in \(y\) into a part explained by the line and a part left over.
Theorem 2.9.1 (Fundamental identity). \[ \underbrace {\sum (y_i - \bar y)^2}_{SS_T} = \underbrace {\sum (\hat y_i - \bar y)^2}_{SS_R} + \underbrace {\sum (y_i - \hat y_i)^2}_{SS_{\mathrm {Res}}} , \] and \(SS_R = \hat \beta _1 S_{xy} = \hat \beta _1^2 S_{xx}\).
Proof. Write \(y_i - \bar y = (\hat y_i - \bar y) + e_i\), square and sum: \[ SS_T = SS_R + SS_{\mathrm {Res}} + 2\sum (\hat y_i - \bar y) e_i . \] The cross term is \(2\sum \hat y_i e_i - 2\bar y \sum e_i = 0\) by Theorem 2.3.1(i) and (iv). For the second statement, \(\hat y_i - \bar y = \hat \beta _0 + \hat \beta _1 x_i - \bar y = \hat \beta _1 (x_i - \bar x)\), so \(SS_R = \hat \beta _1^2 S_{xx} = \hat \beta _1 (\hat \beta _1 S_{xx}) = \hat \beta _1 S_{xy}\). □
\(SS_T = S_{yy}\) is the total sum of squares, with \(n-1\) degrees of freedom; \(SS_R\) is the regression sum of squares, with \(1\) degree of freedom (one slope); and \(SS_{\mathrm {Res}}\) has the remaining \(n - 2\). Each mean square is a sum of squares divided by its degrees of freedom: \(MS_R = SS_R/1\) and \(MS_{\mathrm {Res}} = SS_{\mathrm {Res}}/(n-2)\).
Theorem 2.9.2. \(E(MS_R) = \sigma ^2 + \beta _1^2 S_{xx}\) and \(E(MS_{\mathrm {Res}}) = \sigma ^2\). When \(\beta _1 = 0\), \[ F_0 = \frac {MS_R}{MS_{\mathrm {Res}}} \sim F_{1, n-2} . \] Moreover \(F_0 = t_0^2\), where \(t_0 = \hat \beta _1 / \operatorname {se}(\hat \beta _1)\).
Proof. \(E(MS_R) = S_{xx} E(\hat \beta _1^2) = S_{xx}\left ( \frac {\sigma ^2}{S_{xx}} + \beta _1^2 \right ) = \sigma ^2 + \beta _1^2 S_{xx}\), as in the proof of Theorem 2.5.3, which also gives \(E(MS_{\mathrm {Res}}) = \sigma ^2\).
When \(\beta _1 = 0\), \(\hat \beta _1 \sqrt {S_{xx}}/\sigma \sim N(0,1)\), so \(SS_R/\sigma ^2 = \hat \beta _1^2 S_{xx}/\sigma ^2 \sim \chi ^2_1\). It is a function of \(\hat \beta _1\), so it is independent of \(SS_{\mathrm {Res}}/\sigma ^2 \sim \chi ^2_{n-2}\) (Theorem 2.7.3). By (F3), \[ \frac {(SS_R/\sigma ^2)/1}{(SS_{\mathrm {Res}}/\sigma ^2)/(n-2)} = \frac {MS_R}{MS_{\mathrm {Res}}} \sim F_{1, n-2} . \] Finally \(F_0 = \dfrac {\hat \beta _1^2 S_{xx}}{\hat \sigma ^2} = \left ( \dfrac {\hat \beta _1}{\sqrt {\hat \sigma ^2/S_{xx}}} \right )^2 = t_0^2\). □
The expected mean squares explain the test. When \(\beta _1 = 0\) both mean squares estimate \(\sigma ^2\) and \(F_0\) should be near \(1\); when \(\beta _1 \neq 0\), \(MS_R\) is inflated by \(\beta _1^2 S_{xx}\) and \(F_0\) tends to be large. So reject \(H_0\colon \beta _1 = 0\) when \(F_0 > F_{\alpha , 1, n-2}\). The working is set out in an analysis of variance (ANOVA) table:
| Source | Sum of squares | df | Mean square | \(F_0\) |
| Regression | \(SS_R = \hat \beta _1 S_{xy}\) | \(1\) | \(MS_R\) | \(MS_R / MS_{\mathrm {Res}}\) |
| Residual | \(SS_{\mathrm {Res}} = SS_T - SS_R\) | \(n-2\) | \(MS_{\mathrm {Res}}\) | |
| Total | \(SS_T = S_{yy}\) | \(n-1\) | ||
Example 2.9.3. Set out the ANOVA table for the advertising data and test significance of regression at the 5% level.
Solution. \(SS_T = S_{yy} = 6\), \(SS_R = \hat \beta _1 S_{xy} = 0.7 \times 7 = 4.9\) and \(SS_{\mathrm {Res}} = 6 - 4.9 = 1.1\).
| Source | SS | df | MS | \(F_0\) |
| Regression | 4.9 | 1 | 4.9 | 13.36 |
| Residual | 1.1 | 3 | 0.3667 | |
| Total | 6.0 | 4 | ||
From tables, \(F_{0.05,1,3} = 10.13\). Since \(13.36 > 10.13\), reject \(H_0\colon \beta _1 = 0\). This is the same conclusion as the \(t\)-test of Example 2.8.2, as it must be: \(t_0^2 = 3.656^2 = 13.36 = F_0\), and \(t_{0.025,3}^2 = 3.182^2 = 10.13 = F_{0.05,1,3}\).
Remark. In simple regression the two tests are the same test. The \(t\)-test is more flexible: it can be one-sided, and can test \(\beta _1\) against values other than \(0\). The \(F\)-test is the one that generalises to several regressors at once, in multiple regression.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.