3.8 Test for significance of regression
The first question about a fitted model is whether it is of any use at all: is there a linear relationship between \(y\) and any of the regressors? The hypotheses are \[ H_0\colon \beta _1 = \beta _2 = \cdots = \beta _k = 0 \qquad \text {against}\qquad H_1\colon \beta _j \neq 0 \text { for at least one } j . \] The intercept is not included: \(H_0\) says that \(E(y) = \beta _0\) whatever the regressors are.
As in simple regression, the total sum of squares splits as \(SS_T = SS_R + SS_{\mathrm {Res}}\) (Problem 3.13.5), where \begin {align*} SS_T &= \mathbf {y}^\top \mathbf {y} - \frac {\left (\sum y_i\right )^2}{n}, \\ SS_R &= \hat {\boldsymbol \beta }^\top \mathbf {X}^\top \mathbf {y} - \frac {\left (\sum y_i\right )^2}{n}, \\ SS_{\mathrm {Res}} &= \mathbf {y}^\top \mathbf {y} - \hat {\boldsymbol \beta }^\top \mathbf {X}^\top \mathbf {y} . \end {align*}
\(SS_R\) has \(k\) degrees of freedom, one for each slope, and \(SS_{\mathrm {Res}}\) has \(n - p\).
Theorem 3.8.1. Under A1–A4, when \(H_0\colon \beta _1 = \cdots = \beta _k = 0\) is true, \[ F_0 = \frac {SS_R/k}{SS_{\mathrm {Res}}/(n-p)} = \frac {MS_R}{MS_{\mathrm {Res}}} \sim F_{k, n-p} . \] When \(H_0\) is false, \(E(SS_R) > k\sigma ^2\), so large values of \(F_0\) are evidence against \(H_0\).
Proof. Let \(\mathbf {1}\) be the vector of ones and \(\bar {\mathbf {J}} = \frac 1n\mathbf {1}\mathbf {1}^\top \), the \(n \times n\) matrix with every entry \(1/n\). Then \(\mathbf {y}^\top \bar {\mathbf {J}}\mathbf {y} = \left (\sum y_i\right )^2/n\), and \(\mathbf {y}^\top \mathbf {H}\mathbf {y} = \hat {\boldsymbol \beta }^\top \mathbf {X}^\top \mathbf {y}\) (proof of Theorem 3.6.1), so \(SS_R = \mathbf {y}^\top (\mathbf {H} - \bar {\mathbf {J}})\mathbf {y}\).
Properties of \(\mathbf {H} - \bar {\mathbf {J}}\). \(\mathbf {1}\) is the first column of \(\mathbf {X}\), so \(\mathbf {H}\mathbf {1} = \mathbf {1}\) by Theorem 3.4.2(ii). Hence \(\mathbf {H}\bar {\mathbf {J}} = \bar {\mathbf {J}}\), and transposing, \(\bar {\mathbf {J}}\mathbf {H} = \bar {\mathbf {J}}\). Also \(\bar {\mathbf {J}}^2 = \frac {1}{n^2}\mathbf {1}(\mathbf {1}^\top \mathbf {1})\mathbf {1}^\top = \bar {\mathbf {J}}\). So \begin {align*} (\mathbf {H} - \bar {\mathbf {J}})^2 &= \mathbf {H} - \bar {\mathbf {J}} - \bar {\mathbf {J}} + \bar {\mathbf {J}} = \mathbf {H} - \bar {\mathbf {J}}, \\ (\mathbf {H} - \bar {\mathbf {J}})(\mathbf {I} - \mathbf {H}) &= \mathbf {H} - \mathbf {H} - \bar {\mathbf {J}} + \bar {\mathbf {J}} = \mathbf {0}, \end {align*}
and \(\operatorname {tr}(\mathbf {H} - \bar {\mathbf {J}}) = p - 1 = k\). It is symmetric and idempotent of rank \(k\).
Under \(H_0\), \(\mathbf {X}\boldsymbol \beta = \beta _0\mathbf {1}\), and \((\mathbf {H} - \bar {\mathbf {J}})\mathbf {1} = \mathbf {1} - \mathbf {1} = \mathbf {0}\). So \((\mathbf {H} - \bar {\mathbf {J}})\mathbf {y} = (\mathbf {H} - \bar {\mathbf {J}})\boldsymbol \varepsilon \) and \(SS_R = \boldsymbol \varepsilon ^\top (\mathbf {H} - \bar {\mathbf {J}})\boldsymbol \varepsilon \). Since also \(SS_{\mathrm {Res}} = \boldsymbol \varepsilon ^\top (\mathbf {I} - \mathbf {H})\boldsymbol \varepsilon \), Lemma 3.7.1 gives \(SS_R/\sigma ^2 \sim \chi ^2_k\) and \(SS_{\mathrm {Res}}/\sigma ^2 \sim \chi ^2_{n-p}\), independently, and (F3) gives the \(F\) distribution.
When \(H_0\) is false. By Lemma 3.1.4, \(E(SS_R) = \sigma ^2\operatorname {tr}(\mathbf {H} - \bar {\mathbf {J}}) + \boldsymbol \mu ^\top (\mathbf {H} - \bar {\mathbf {J}})\boldsymbol \mu \) with \(\boldsymbol \mu = \mathbf {X}\boldsymbol \beta \). Since \(\mathbf {H}\boldsymbol \mu = \boldsymbol \mu \), the second term is \(\boldsymbol \mu ^\top (\mathbf {I} - \bar {\mathbf {J}})\boldsymbol \mu = \sum (\mu _i - \bar \mu )^2\), which is positive unless all the \(\mu _i\) are equal. With \(\mathbf {X}\) of full rank that happens only when \(\beta _1 = \cdots = \beta _k = 0\). □
Reject \(H_0\) at level \(\alpha \) when \(F_0 > F_{\alpha , k, n-p}\). The working goes in an ANOVA table:
| Source | Sum of squares | df | Mean square | \(F_0\) |
| Regression | \(SS_R\) | \(k\) | \(MS_R = SS_R/k\) | \(MS_R/MS_{\mathrm {Res}}\) |
| Residual | \(SS_{\mathrm {Res}}\) | \(n - p\) | \(MS_{\mathrm {Res}}\) | |
| Total | \(SS_T\) | \(n - 1\) | ||
With \(k = 1\) this is exactly the table of Subsection 2.9.
Solution. From Examples 3.3.3 and 3.6.2: \(\sum y_i = 69\), \(\mathbf {y}^\top \mathbf {y} = 887\) and \(\hat {\boldsymbol \beta }^\top \mathbf {X}^\top \mathbf {y} = 883\). The correction term is \(69^2/6 = 793.5\), so \begin {align*} SS_T &= 887 - 793.5 = 93.5, \\ SS_R &= 883 - 793.5 = 89.5, \\ SS_{\mathrm {Res}} &= 887 - 883 = 4 . \end {align*}
| Source | SS | df | MS | \(F_0\) |
| Regression | 89.5 | 2 | 44.75 | 33.56 |
| Residual | 4.0 | 3 | 1.3333 | |
| Total | 93.5 | 5 | ||
\(F_{0.05,2,3} = 9.55\), and \(33.56 > 9.55\), so reject \(H_0\colon \beta _1 = \beta _2 = 0\) (\(p = 0.009\)). Yield is linearly related to at least one of fertiliser and rainfall.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.