2.7 Sampling distributions under normality
To test hypotheses and build confidence intervals we need the distributions of \(\hat \beta _0\), \(\hat \beta _1\) and \(SS_{\mathrm {Res}}\), not just their means and variances. From here to the end of this section, A1–A4 are assumed.
We use three facts from distribution theory:
- (F1)
- A linear combination of independent normal variables is normal. More generally, variables that are all linear combinations of the same independent normal variables are jointly normal, and jointly normal variables that are uncorrelated are independent.
- (F2)
- If \(Z \sim N(0,1)\) then \(Z^2 \sim \chi ^2_1\). The \(\chi ^2_\nu \) distribution has moment generating function \((1 - 2t)^{-\nu /2}\), and the sum of independent \(\chi ^2\) variables is \(\chi ^2\) with the degrees of freedom added.
- (F3)
- If \(Z \sim N(0,1)\) and \(V \sim \chi ^2_\nu \) are independent, then \(Z \big / \sqrt {V/\nu } \sim t_\nu \); and if \(V_1 \sim \chi ^2_{\nu _1}\) and \(V_2 \sim \chi ^2_{\nu _2}\) are independent, then \(\dfrac {V_1/\nu _1}{V_2/\nu _2} \sim F_{\nu _1, \nu _2}\).
Theorem 2.7.1. \(\hat \beta _1 \sim N\!\left (\beta _1, \dfrac {\sigma ^2}{S_{xx}}\right )\) and \(\hat \beta _0 \sim N\!\left (\beta _0, \sigma ^2\left (\dfrac 1n + \dfrac {\bar x^2}{S_{xx}}\right )\right )\).
Proof. Both are linear combinations of the independent normal \(y_i\) (Lemma 2.4.1, and \(\hat \beta _0 = \bar y - \bar x \hat \beta _1\)), so both are normal by (F1). Their means and variances are those of Theorems 2.4.2 and 2.4.4. □
Lemma 2.7.2. The residuals \(e_1, \dots , e_n\) are independent of the pair \((\bar y, \hat \beta _1)\), and \(\bar y\) is independent of \(\hat \beta _1\).
Proof. Every \(e_i\), together with \(\bar y\) and \(\hat \beta _1\), is a linear combination of the independent normal \(y_j\), so by (F1) it is enough to show that the covariances are zero. Lemma 2.4.3 gives \(\operatorname {Cov}(\bar y, \hat \beta _1) = 0\). Next, \(e_i = y_i - \bar y - \hat \beta _1 (x_i - \bar x)\). From \(\hat \beta _1 = \sum c_j y_j\), \(\operatorname {Cov}(y_i, \hat \beta _1) = c_i \sigma ^2\), and from \(\bar y = \sum y_j/n\), \(\operatorname {Cov}(y_i, \bar y) = \sigma ^2/n\). So \begin {align*} \operatorname {Cov}(e_i, \hat \beta _1) &= c_i\sigma ^2 - 0 - (x_i - \bar x)\frac {\sigma ^2}{S_{xx}} = \frac {(x_i - \bar x)\sigma ^2}{S_{xx}} - \frac {(x_i - \bar x)\sigma ^2}{S_{xx}} = 0, \\ \operatorname {Cov}(e_i, \bar y) &= \frac {\sigma ^2}{n} - \frac {\sigma ^2}{n} - (x_i - \bar x) \cdot 0 = 0 . \qedhere \end {align*} □
Theorem 2.7.3. \(\dfrac {SS_{\mathrm {Res}}}{\sigma ^2} \sim \chi ^2_{n-2}\), and \(SS_{\mathrm {Res}}\) is independent of \(\hat \beta _0\) and \(\hat \beta _1\).
Proof. \(SS_{\mathrm {Res}} = \sum e_i^2\) is a function of the residuals alone, and \(\hat \beta _0\) and \(\hat \beta _1\) are functions of \((\bar y, \hat \beta _1)\), so the independence follows from Lemma 2.7.2.
For the distribution, split each error into three parts. Since \(\hat \beta _0 = \bar y - \hat \beta _1 \bar x\), \[ \varepsilon _i = y_i - \beta _0 - \beta _1 x_i = e_i + (\bar y - \beta _0 - \beta _1 \bar x) + (\hat \beta _1 - \beta _1)(x_i - \bar x) , \] as can be checked by expanding \(e_i = y_i - \bar y - \hat \beta _1(x_i - \bar x)\). Square and sum over \(i\). All three cross terms vanish, because \(\sum e_i = 0\), \(\sum (x_i - \bar x) e_i = \sum x_i e_i - \bar x \sum e_i = 0\) (Theorem 2.3.1) and \(\sum (x_i - \bar x) = 0\). So \[ \sum \varepsilon _i^2 = SS_{\mathrm {Res}} + n(\bar y - \beta _0 - \beta _1 \bar x)^2 + (\hat \beta _1 - \beta _1)^2 S_{xx} . \] Divide by \(\sigma ^2\) and look at each term.
- Left side: \(\varepsilon _i/\sigma \sim N(0,1)\) independently, so \(\sum \varepsilon _i^2/\sigma ^2 \sim \chi ^2_n\) by (F2).
- \(\bar y \sim N(\beta _0 + \beta _1 \bar x, \sigma ^2/n)\), so \(n(\bar y - \beta _0 - \beta _1 \bar x)^2/\sigma ^2 = Z_1^2 \sim \chi ^2_1\).
- By Theorem 2.7.1, \((\hat \beta _1 - \beta _1)^2 S_{xx}/\sigma ^2 = Z_2^2 \sim \chi ^2_1\).
By Lemma 2.7.2 the three terms on the right are independent. Taking moment generating functions, with \(M(t)\) that of \(SS_{\mathrm {Res}}/\sigma ^2\), \[ (1 - 2t)^{-n/2} = M(t)\,(1 - 2t)^{-1/2}(1 - 2t)^{-1/2}, \] so that \[ M(t) = (1 - 2t)^{-(n-2)/2}, \] which is the moment generating function of \(\chi ^2_{n-2}\). □
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.