2.4 Inference
Under the normality assumption (A5), \(b = \beta + \sum w_iu_i\) is a linear combination of independent normal variables, so \[ b \sim N\!\left (\beta ,\ \frac {\sigma ^2}{\sum x_i^2}\right ), \qquad \text {that is}\qquad \frac {b - \beta }{\sigma /\sqrt {\sum x_i^2}} \sim N(0, 1) . \] Replacing \(\sigma \) by \(s\) changes the distribution, because of two further facts: \[ \frac {(n-2)s^2}{\sigma ^2} \sim \chi ^2_{n-2}, \qquad s^2 \text { is independent of } (a, b) . \] Both are consequences of the theory of quadratic forms in normal variables, proved for the general linear model with the theory of quadratic forms. Granting them, the ratio of a standard normal to the square root of an independent \(\chi ^2\) over its degrees of freedom is Student’s \(t\):
Theorem 2.4.1. Under (A1)–(A5), \[ \frac {b - \beta }{\operatorname {se}(b)} \sim t_{n-2}, \qquad \frac {a - \alpha }{\operatorname {se}(a)} \sim t_{n-2} . \]
Proof. \[ \frac {b - \beta }{\operatorname {se}(b)} = \frac {(b - \beta )\big /\bigl (\sigma /\sqrt {\sum x_i^2}\bigr )}{\sqrt {\bigl [(n-2)s^2/\sigma ^2\bigr ]\big /(n-2)}} , \] a standard normal divided by the square root of an independent \(\chi ^2_{n-2}\) over \(n - 2\). The argument for \(a\) is the same. □
Three standard procedures follow, with \(t_{\alpha /2}\) the upper \(\alpha /2\) point of \(t_{n-2}\).
- (i)
- Confidence interval: \(b \pm t_{\alpha /2}\operatorname {se}(b)\) covers \(\beta \) with probability \(1 - \alpha \).
- (ii)
- Test of \(H_0\colon \beta = \beta _0\): reject at level \(\alpha \) against \(\beta \neq \beta _0\) if \(|b - \beta _0|/\operatorname {se}(b) > t_{\alpha /2}\). The interval in (i) is exactly the set of \(\beta _0\) that this test does not reject.
- (iii)
- Test of the regression: \(H_0\colon \beta = 0\), that \(X\) has no linear effect on \(Y\). Its \(t\) ratio is the one every regression program prints.
Theorem 2.4.2. The statistic \[ F = \frac {\text {ESS}/1}{\text {RSS}/(n-2)} \] is the square of the \(t\) ratio for \(H_0\colon \beta = 0\), and under \(H_0\) it has the \(F_{1,\,n-2}\) distribution. It is also equal to \(\dfrac {r^2(n-2)}{1 - r^2}\), the square of the \(t\) statistic for zero correlation used in Section 1.5.
Proof. \(\text {ESS} = b^2\sum x_i^2\) and \(\text {RSS}/(n-2) = s^2\), so \(F = b^2\sum x_i^2/s^2 = \bigl (b/\operatorname {se}(b)\bigr )^2\). The square of a \(t_{n-2}\) variable is \(F_{1,n-2}\). Dividing numerator and denominator by TSS, \(F = (n-2)R^2/(1 - R^2)\), and \(R^2 = r^2\) by Theorem 2.2.4. □
Example 2.4.3. In Example 2.2.5, \(s^2 = 1.6/3 = 0.5333\), so \[ \operatorname {se}(b) = \sqrt {0.5333/40} = 0.1155, \qquad t = \frac {0.7}{0.1155} = 6.06 . \] With \(3\) degrees of freedom, \(t_{0.025} = 3.182\). The hypothesis \(\beta = 0\) is rejected at the \(5\%\) level, and the \(95\%\) confidence interval is \[ 0.7 \pm 3.182(0.1155) = 0.7 \pm 0.367 = (0.333,\ 1.067) . \] The interval is wide, as it must be with five observations, but it excludes zero. As a check on Theorem 2.4.2, \(F = 19.6/0.5333 = 36.75 = 6.06^2\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.