3.6 Estimation of \(\sigma ^2\) in multiple regression

Theorem 3.6.1.

(i)
\(SS_{\mathrm {Res}} = \mathbf {e}^\top \mathbf {e} = \mathbf {y}^\top (\mathbf {I} - \mathbf {H})\mathbf {y} = \mathbf {y}^\top \mathbf {y} - \hat {\boldsymbol \beta }^\top \mathbf {X}^\top \mathbf {y}\).
(ii)
Under A1–A3, \(E(SS_{\mathrm {Res}}) = (n - p)\sigma ^2\), so \[ \hat \sigma ^2 = MS_{\mathrm {Res}} = \frac {SS_{\mathrm {Res}}}{n - p} \] is unbiased for \(\sigma ^2\).

Proof. (i) By Theorem 3.4.2(i), \(\mathbf {e}^\top \mathbf {e} = \mathbf {y}^\top (\mathbf {I} - \mathbf {H})^\top (\mathbf {I} - \mathbf {H})\mathbf {y} = \mathbf {y}^\top (\mathbf {I} - \mathbf {H})\mathbf {y}\). And \(\mathbf {y}^\top \mathbf {H}\mathbf {y} = \mathbf {y}^\top \mathbf {X}\mathbf {C}\mathbf {X}^\top \mathbf {y} = \mathbf {y}^\top \mathbf {X}\hat {\boldsymbol \beta } = \hat {\boldsymbol \beta }^\top \mathbf {X}^\top \mathbf {y}\).

(ii) By Lemma 3.1.4 with \(\mathbf {A} = \mathbf {I} - \mathbf {H}\), \(\boldsymbol \mu = \mathbf {X}\boldsymbol \beta \) and \(\mathbf {V} = \sigma ^2\mathbf {I}\), \[ E(SS_{\mathrm {Res}}) = \sigma ^2\operatorname {tr}(\mathbf {I} - \mathbf {H}) + \boldsymbol \beta ^\top \mathbf {X}^\top (\mathbf {I} - \mathbf {H})\mathbf {X}\boldsymbol \beta = \sigma ^2(n - p) + 0 , \] by Theorem 3.4.2(iv) and (ii). □

With \(k = 1\), \(p = 2\) and this is Theorem 2.5.3, proved now in two lines. One degree of freedom is lost for each estimated coefficient. The standard error of \(\hat \beta _j\) is \[ \operatorname {se}(\hat \beta _j) = \sqrt {\hat \sigma ^2 C_{jj}} . \]

Example 3.6.2. Estimate \(\sigma ^2\) for the crop data, and find the standard errors of the coefficients.

Solution. From the residuals, \(SS_{\mathrm {Res}} = 1 + 1 + 1 + 1 + 0 + 0 = 4\). Check with Theorem 3.6.1(i): \(\mathbf {y}^\top \mathbf {y} = \sum y_i^2 = 887\) and \[ \hat {\boldsymbol \beta }^\top \mathbf {X}^\top \mathbf {y} = 3(69) + 2(244) + 1(188) = 207 + 488 + 188 = 883, \] so \(SS_{\mathrm {Res}} = 887 - 883 = 4\). With \(n - p = 6 - 3 = 3\), \[ \hat \sigma ^2 = \frac 43 = 1.3333 . \] The diagonal of \(\mathbf {C}\) is \(C_{00} = 510/378\), \(C_{11} = 33/378\), \(C_{22} = 96/378\), so \begin {align*} \operatorname {se}(\hat \beta _0) &= \sqrt {1.3333 \times 510/378} = 1.341, \\ \operatorname {se}(\hat \beta _1) &= \sqrt {1.3333 \times 33/378} = 0.341, \\ \operatorname {se}(\hat \beta _2) &= \sqrt {1.3333 \times 96/378} = 0.582 . \end {align*}

The estimated covariance of \(\hat \beta _1\) and \(\hat \beta _2\) is \(\hat \sigma ^2 C_{12} = 1.3333 \times (-30/378) = -0.106\). It is negative because, in these data, fertiliser and rainfall tend to rise together: credit for a high yield can be shifted from one to the other, so an overestimate of one coefficient tends to go with an underestimate of the other.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.