3.7 Distribution theory under normality
From here on A4 is assumed: \(\boldsymbol \varepsilon \sim N_n(\mathbf {0}, \sigma ^2\mathbf {I})\). The distributional facts (F1)–(F3) of Subsection 2.7 are used again, together with one result from linear algebra, the spectral theorem: a symmetric matrix \(\mathbf {A}\) can be written \(\mathbf {A} = \mathbf {P}\boldsymbol \Lambda \mathbf {P}^\top \), where \(\boldsymbol \Lambda \) is the diagonal matrix of its eigenvalues and \(\mathbf {P}\) is orthogonal, \(\mathbf {P}^\top \mathbf {P} = \mathbf {I}\).
Lemma 3.7.1. Let \(\mathbf {z} \sim N_n(\mathbf {0}, \sigma ^2\mathbf {I})\) and let \(\mathbf {A}\) be symmetric and idempotent.
- (i)
- The eigenvalues of \(\mathbf {A}\) are \(0\) or \(1\), and the number equal to \(1\) is \(r = \operatorname {tr}(\mathbf {A})\), the rank of \(\mathbf {A}\).
- (ii)
- \(\mathbf {z}^\top \mathbf {A}\mathbf {z}/\sigma ^2 \sim \chi ^2_r\).
- (iii)
- If \(\mathbf {B}\) is also symmetric and idempotent and \(\mathbf {AB} = \mathbf {0}\), then \(\mathbf {z}^\top \mathbf {A}\mathbf {z}\) and \(\mathbf {z}^\top \mathbf {B}\mathbf {z}\) are independent.
Proof. (i) If \(\mathbf {Av} = \lambda \mathbf {v}\) with \(\mathbf {v} \neq \mathbf {0}\), then \(\lambda \mathbf {v} = \mathbf {Av} = \mathbf {A}^2\mathbf {v} = \lambda ^2\mathbf {v}\), so \(\lambda ^2 = \lambda \) and \(\lambda \) is \(0\) or \(1\). The trace of \(\mathbf {P}\boldsymbol \Lambda \mathbf {P}^\top \) is \(\operatorname {tr}(\boldsymbol \Lambda \mathbf {P}^\top \mathbf {P}) = \operatorname {tr}(\boldsymbol \Lambda )\) by Lemma 3.1.1, the sum of the eigenvalues, which here is the number of ones.
(ii) Let \(\mathbf {w} = \mathbf {P}^\top \mathbf {z}/\sigma \). By Lemma 3.1.3 it is normal with mean \(\mathbf {0}\) and covariance \(\mathbf {P}^\top (\sigma ^2\mathbf {I})\mathbf {P}/\sigma ^2 = \mathbf {I}\), so its entries are independent \(N(0,1)\). Then \[ \frac {\mathbf {z}^\top \mathbf {A}\mathbf {z}}{\sigma ^2} = \mathbf {w}^\top \mathbf {P}^\top \mathbf {A}\mathbf {P}\mathbf {w} = \mathbf {w}^\top \boldsymbol \Lambda \mathbf {w} = \sum _{i\colon \lambda _i = 1} w_i^2 , \] a sum of \(r\) squared independent standard normals, which is \(\chi ^2_r\) by (F2).
(iii) \(\mathbf {Az}\) and \(\mathbf {Bz}\) are jointly normal, with \(\operatorname {Cov}(\mathbf {Az}, \mathbf {Bz}) = \mathbf {A}(\sigma ^2\mathbf {I})\mathbf {B}^\top = \sigma ^2\mathbf {AB} = \mathbf {0}\), so they are independent by (F1). And \(\mathbf {z}^\top \mathbf {A}\mathbf {z} = \mathbf {z}^\top \mathbf {A}^\top \mathbf {A}\mathbf {z} = (\mathbf {Az})^\top (\mathbf {Az})\) is a function of \(\mathbf {Az}\) alone; likewise \(\mathbf {z}^\top \mathbf {B}\mathbf {z}\) of \(\mathbf {Bz}\). □
- (i)
- \(\hat {\boldsymbol \beta } \sim N_p\bigl (\boldsymbol \beta , \sigma ^2\mathbf {C}\bigr )\), so each \(\hat \beta _j \sim N(\beta _j, \sigma ^2 C_{jj})\);
- (ii)
- \(SS_{\mathrm {Res}}/\sigma ^2 \sim \chi ^2_{n-p}\);
- (iii)
- \(SS_{\mathrm {Res}}\) is independent of \(\hat {\boldsymbol \beta }\).
Proof. (i) \(\hat {\boldsymbol \beta }\) is linear in \(\mathbf {y}\), hence normal by (F1), with the mean and covariance of Theorem 3.5.1.
(ii) \((\mathbf {I} - \mathbf {H})\mathbf {y} = (\mathbf {I} - \mathbf {H})\boldsymbol \varepsilon \) because \((\mathbf {I} - \mathbf {H})\mathbf {X} = \mathbf {0}\), so \(SS_{\mathrm {Res}} = \boldsymbol \varepsilon ^\top (\mathbf {I} - \mathbf {H})\boldsymbol \varepsilon \). \(\mathbf {I} - \mathbf {H}\) is symmetric and idempotent with trace \(n - p\) (Theorem 3.4.2), so Lemma 3.7.1(ii) applies.
(iii) \(\hat {\boldsymbol \beta } - \boldsymbol \beta = \mathbf {C}\mathbf {X}^\top \boldsymbol \varepsilon \) and \(\mathbf {e} = (\mathbf {I} - \mathbf {H})\boldsymbol \varepsilon \) are jointly normal, with \[ \operatorname {Cov}\bigl (\mathbf {C}\mathbf {X}^\top \boldsymbol \varepsilon ,\ (\mathbf {I} - \mathbf {H})\boldsymbol \varepsilon \bigr ) = \sigma ^2\mathbf {C}\bigl ((\mathbf {I} - \mathbf {H})\mathbf {X}\bigr )^\top = \mathbf {0} . \] So \(\hat {\boldsymbol \beta }\) is independent of \(\mathbf {e}\), and hence of \(SS_{\mathrm {Res}} = \mathbf {e}^\top \mathbf {e}\). □
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.