5.1 Definition and Basic Properties
The distribution this section is named after has been used implicitly since the covariance matrices of Section 2; it is defined here, and the properties that the inference of the following subsections relies on are established before they are used.
Definition 5.1 (Multivariate normal distribution). A random vector \(\underline {X} = (X_1,\dots ,X_p)'\) has the multivariate normal distribution with mean \(\underline {\mu }\) and covariance matrix \(\Sigma \), written \(\underline {X}\sim N_p(\underline {\mu },\Sigma )\), if for \(\Sigma \) positive definite its density is \[f(\underline {x}) = \frac {1}{(2\pi )^{p/2}\left |\Sigma \right |^{1/2}} \exp \left \{-\tfrac {1}{2}\left (\underline {x}-\underline {\mu }\right )' \Sigma ^{-1}\left (\underline {x}-\underline {\mu }\right )\right \}, \qquad \underline {x}\in \mathbb {R}^{p}.\]
Note 5.2. Compare the exponent with the univariate case, where it is \(-\left (x-\mu \right )^{2}/2\sigma ^{2}\). The quadratic form \(\left (\underline {x}-\underline {\mu }\right )'\Sigma ^{-1} \left (\underline {x}-\underline {\mu }\right )\) is the squared Mahalanobis distance of Section 1.9, so the density depends on \(\underline {x}\) only through that distance from the mean — it is constant on the ellipsoids of Section 4, whose axes are the eigenvectors of \(\Sigma \) and whose extents are proportional to \(\sqrt {\lambda _i}\). The two constants have the same origin as in one dimension: \(\left |\Sigma \right |^{1/2}\) generalises \(\sigma \), and \((2\pi )^{p/2}\) is the normalising constant of \(p\) independent standard normals.
Theorem 5.3 (Linear combinations). If \(\underline {X}\sim N_p(\underline {\mu },\Sigma )\) then for any \(A_{q\times p}\) and \(\underline {b}_{q\times 1}\) of constants, \[A\underline {X}+\underline {b}\ \sim \ N_q\left (A\underline {\mu }+\underline {b},\ A\Sigma A'\right ).\] In particular every linear combination \(\underline {a}'\underline {X}\) is univariate normal.
Note 5.4. Theorem 5.3 is the property that makes the multivariate normal tractable, and it has a converse that is often taken as the definition: \(\underline {X}\) is multivariate normal precisely when \(\underline {a}'\underline {X}\) is univariate normal for every fixed \(\underline {a}\). That formulation has the advantage of remaining meaningful when \(\Sigma \) is singular, where no density exists — the distribution then lives on a lower-dimensional subspace, which is exactly the situation met in Section 4 when an eigenvalue is zero.
Theorem 5.5 (Marginals and independence). Partition \(\underline {X} = \begin {pmatrix}\underline {X}_1\\ \underline {X}_2\end {pmatrix}\) conformably with \(\underline {\mu } = \begin {pmatrix}\underline {\mu }_1\\ \underline {\mu }_2\end {pmatrix}\) and \(\Sigma = \begin {pmatrix}\Sigma _{11} & \Sigma _{12}\\ \Sigma _{21} & \Sigma _{22}\end {pmatrix}\). Then
- (i)
- \(\underline {X}_1 \sim N_{q}\left (\underline {\mu }_1,\Sigma _{11}\right )\);
- (ii)
- \(\underline {X}_1\) and \(\underline {X}_2\) are independent if and only if \(\Sigma _{12}=0\).
Note 5.6. Part (ii) deserves emphasis because it is false in general. Zero covariance does not imply independence for arbitrary distributions — it says only that no linear association is present. Under joint normality the two coincide, and this equivalence is used constantly in what follows. It requires joint normality: two variables that are each normal but not jointly so can be uncorrelated and dependent.
Theorem 5.7 (Conditional distributions). With the partition above and \(\Sigma _{22}\) non-singular, \[\underline {X}_1 \mid \underline {X}_2 = \underline {x}_2\ \sim \ N_q\left (\underline {\mu }_1 + \Sigma _{12}\Sigma _{22}^{-1} \left (\underline {x}_2-\underline {\mu }_2\right ),\ \ \Sigma _{11}-\Sigma _{12}\Sigma _{22}^{-1}\Sigma _{21}\right ).\]
Note 5.8. Two features of Theorem 5.7 are worth naming. The conditional mean is a linear function of \(\underline {x}_2\) — which is where the linear regression of Section 8 comes from, the regression coefficients being \(\Sigma _{12}\Sigma _{22}^{-1}\). And the conditional covariance does not depend on \(\underline {x}_2\) at all: the conditional variability is the same wherever one conditions, which is the multivariate statement of homoscedasticity. Both are special properties of the normal and neither holds in general.
The matrix \(\Sigma _{11}-\Sigma _{12}\Sigma _{22}^{-1}\Sigma _{21}\) is the Schur complement of \(\Sigma _{22}\), and it is never larger than \(\Sigma _{11}\) in the positive-definite ordering: conditioning on information cannot increase uncertainty.
Theorem 5.9 (The quadratic form). If \(\underline {X}\sim N_p(\underline {\mu },\Sigma )\) with \(\Sigma \) non-singular, then \[\left (\underline {X}-\underline {\mu }\right )'\Sigma ^{-1} \left (\underline {X}-\underline {\mu }\right )\ \sim \ \chi ^{2}_{p}.\]
Proof. By Section 1.9 write \(\Sigma ^{-1/2}\) for the symmetric square root and put \(\underline {Z} = \Sigma ^{-1/2}\left (\underline {X}-\underline {\mu }\right )\). By Theorem 5.3, \(\underline {Z}\sim N_p(\underline {0}, I)\), so its components are independent standard normals, and the quadratic form equals \(\underline {Z}'\underline {Z} = \sum ^{p}_{i=1}Z_i^{2}\), a sum of \(p\) independent squared standard normals. □
Note 5.10. Theorem 5.9 is what makes the confidence regions of the following subsections ellipsoids rather than boxes. The set of \(\underline {x}\) with Mahalanobis distance below the \(\chi ^{2}_{p}\) critical value is an ellipsoid centred at \(\underline {\mu }\), oriented along the eigenvectors of \(\Sigma \); it is also the natural region for detecting outliers, since a point may be unremarkable in each coordinate separately and far away in the joint sense.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.