2.5 Estimation of \(\sigma ^2\)
The variances in Theorems 2.4.2 and 2.4.4 involve \(\sigma ^2\), which is unknown. It is estimated from the residuals.
Definition 2.5.1. The residual sum of squares is \[ SS_{\mathrm {Res}} = \sum _{i=1}^n e_i^2 = \sum _{i=1}^n (y_i - \hat y_i)^2 . \]
Proof. Since \(\hat \beta _0 = \bar y - \hat \beta _1 \bar x\), each residual can be written \(e_i = y_i - \bar y + \hat \beta _1 \bar x - \hat \beta _1 x_i = (y_i - \bar y) - \hat \beta _1 (x_i - \bar x)\). Squaring and summing, \[ SS_{\mathrm {Res}} = S_{yy} - 2\hat \beta _1 S_{xy} + \hat \beta _1^2 S_{xx} . \] But \(\hat \beta _1 S_{xx} = S_{xy}\), so \(\hat \beta _1^2 S_{xx} = \hat \beta _1 S_{xy}\), and the last two terms combine to \(-\hat \beta _1 S_{xy}\). □
Theorem 2.5.3. Under A1–A3, \(E(SS_{\mathrm {Res}}) = (n-2)\sigma ^2\). Hence \[ \boxed {\;\hat \sigma ^2 = MS_{\mathrm {Res}} = \frac {SS_{\mathrm {Res}}}{n-2}\;} \] is an unbiased estimator of \(\sigma ^2\). It is called the residual mean square.
Proof. We find the expectations of the two terms of Theorem 2.5.2 in turn.
The term \(S_{yy}\). Averaging the model over \(i\) gives \(\bar y = \beta _0 + \beta _1 \bar x + \bar \varepsilon \), so \(y_i - \bar y = \beta _1 (x_i - \bar x) + (\varepsilon _i - \bar \varepsilon )\). Therefore \[ S_{yy} = \beta _1^2 S_{xx} + 2\beta _1 \sum (x_i - \bar x)(\varepsilon _i - \bar \varepsilon ) + \sum (\varepsilon _i - \bar \varepsilon )^2 . \] The middle term has expectation \(0\) by A1. For the last, the identity of Lemma 2.2.3(ii) gives \(\sum (\varepsilon _i - \bar \varepsilon )^2 = \sum \varepsilon _i^2 - n\bar \varepsilon ^2\). By A1 and A2, \(E(\varepsilon _i^2) = \operatorname {Var}(\varepsilon _i) = \sigma ^2\), and by A3 \(\operatorname {Var}(\bar \varepsilon ) = \sigma ^2/n\), so \(E(\bar \varepsilon ^2) = \sigma ^2/n\). Hence \[ E(S_{yy}) = \beta _1^2 S_{xx} + n\sigma ^2 - n\cdot \frac {\sigma ^2}{n} = \beta _1^2 S_{xx} + (n-1)\sigma ^2 . \] The term \(\hat \beta _1 S_{xy}\). This equals \(\hat \beta _1^2 S_{xx}\). For any random variable, \(E(W^2) = \operatorname {Var}(W) + \left (E W\right )^2\), so by Theorem 2.4.2 \[ E(\hat \beta _1 S_{xy}) = S_{xx} \left ( \frac {\sigma ^2}{S_{xx}} + \beta _1^2 \right ) = \sigma ^2 + \beta _1^2 S_{xx} . \] Subtracting, \(E(SS_{\mathrm {Res}}) = \beta _1^2 S_{xx} + (n-1)\sigma ^2 - \sigma ^2 - \beta _1^2 S_{xx} = (n-2)\sigma ^2\). □
Remark. Why divide by \(n - 2\) rather than \(n\)? The \(n\) residuals are not free: they must satisfy the two equations \(\sum e_i = 0\) and \(\sum x_i e_i = 0\) (Theorem 2.3.1), one for each estimated parameter. Only \(n - 2\) of them can vary freely, and \(SS_{\mathrm {Res}}\) is said to have \(n - 2\) degrees of freedom.
Definition 2.5.4. The standard error of an estimator is the estimate of its standard deviation, obtained by putting \(\hat \sigma ^2\) in place of \(\sigma ^2\): \[ \operatorname {se}(\hat \beta _1) = \sqrt {\frac {\hat \sigma ^2}{S_{xx}}}, \qquad \operatorname {se}(\hat \beta _0) = \sqrt {\hat \sigma ^2 \left ( \frac 1n + \frac {\bar x^2}{S_{xx}} \right )} . \]
Example 2.5.5. For the advertising data, estimate \(\sigma ^2\) and find the standard errors of \(\hat \beta _0\) and \(\hat \beta _1\).
Solution. From the residuals of Example 2.2.6, \begin {align*} SS_{\mathrm {Res}} &= 0.4^2 + (-0.3)^2 + 0^2 + (-0.7)^2 + 0.6^2 \\ &= 0.16 + 0.09 + 0 + 0.49 + 0.36 = 1.10 . \end {align*}
As a check, by Theorem 2.5.2: \(S_{yy} = 26 - 10^2/5 = 6\), and \(S_{yy} - \hat \beta _1 S_{xy} = 6 - 0.7 \times 7 = 6 - 4.9 = 1.10\). The two agree. Then \[ \hat \sigma ^2 = \frac {1.10}{5 - 2} = 0.3667, \] and \begin {align*} \operatorname {se}(\hat \beta _1) &= \sqrt {\frac {0.3667}{10}} = 0.1915, \\ \operatorname {se}(\hat \beta _0) &= \sqrt {0.3667 \left ( \frac 15 + \frac {3^2}{10} \right )} = \sqrt {0.3667 \times 1.1} = 0.6351 . \end {align*}
The slope \(0.7\) is more than three times its standard error, which already suggests a real relationship. The test of Subsection 2.8 makes this precise.
Example 2.5.6. Estimate \(\sigma ^2\) for the yield data of Example 2.2.7.
Solution. \(S_{yy} = 26974 - 398^2/6 = 26974 - 26400.6667 = 573.3333\) and \(\hat \beta _1 S_{xy} = (4/7)(1000) = 571.4286\), so \[ SS_{\mathrm {Res}} = 573.3333 - 571.4286 = 1.9048, \qquad \hat \sigma ^2 = \frac {1.9048}{6 - 2} = 0.4762 . \] The short-cut formula subtracts two large, nearly equal numbers, so it needs every decimal place you can keep: with \(\hat \beta _1\) rounded to \(0.57\) it would give \(573.3333 - 570 = 3.33\), nearly double the true value. The residuals themselves are \[ -0.0476, \quad 0.2381, \quad -0.4762, \quad 0.8095, \quad -0.9048, \quad 0.3810, \] and the sum of their squares is again \(1.9048\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.