5.3 The multivariate normal distribution

Definition 5.3.1. An \(n\)-vector \(X\) has the multivariate normal distribution \(N(\mu , \Sigma )\), with \(\Sigma \) pd, if its density is \[ f(x) = (2\pi )^{-n/2}|\Sigma |^{-1/2}\exp \Bigl \{-\tfrac 12(x - \mu )'\Sigma ^{-1}(x - \mu )\Bigr \}, \qquad x \in \mathbb R^n . \]

When \(\Sigma = \sigma ^2I\) the density factorises into \(n\) univariate \(N(\mu _i, \sigma ^2)\) densities, so \(N(\mu , \sigma ^2I)\) is the distribution of \(n\) independent normal variables with common variance. That is the disturbance vector of the classical normal regression model.

Theorem 5.3.2. Let \(X \sim N(\mu , \Sigma )\).

(i)
The moment generating function is \(E(e^{t'X}) = \exp \bigl (t'\mu + \frac 12t'\Sigma t\bigr )\).
(ii)
If \(A\) is \(m \times n\) of rank \(m\), then \(AX + b \sim N(A\mu + b,\ A\Sigma A')\). In particular every subvector of \(X\) is normal.
(iii)
If \(X\) is partitioned into \(X_1\) and \(X_2\), then \(X_1\) and \(X_2\) are independent if and only if \(\Cov (X_1, X_2) = 0\).

Proof. (i) Write \(\Sigma = CC'\) with \(C = \Sigma ^{1/2}\) non-singular, and let \(Z = C^{-1}(X - \mu )\). Changing variables in the density, with Jacobian \(|C|\) and \((x - \mu )'\Sigma ^{-1}(x - \mu ) = z'z\), shows that \(Z\) has density \((2\pi )^{-n/2}e^{-z'z/2}\): its elements are independent \(N(0, 1)\). Each has moment generating function \(e^{s^2/2}\), so \(E(e^{s'Z}) = e^{s's/2}\). Then \(X = \mu + CZ\) and \[ E(e^{t'X}) = e^{t'\mu }E\bigl (e^{(C't)'Z}\bigr ) = e^{t'\mu }e^{t'CC't/2} = \exp \bigl (t'\mu + \tfrac 12t'\Sigma t\bigr ) . \]

(ii) For \(s \in \mathbb R^m\), \(E(e^{s'(AX + b)}) = e^{s'b}E(e^{(A's)'X}) = \exp \bigl (s'(A\mu + b) + \frac 12s'A\Sigma A's\bigr )\), the moment generating function of \(N(A\mu + b, A\Sigma A')\). \(A\Sigma A'\) is pd because \(A\) has full row rank, and the moment generating function determines the distribution. A subvector is \(AX\) for a matrix \(A\) that selects rows of the identity.

(iii) Independence always implies zero covariance. Conversely, if \(\Sigma = \operatorname {diag}(\Sigma _{11}, \Sigma _{22})\), then \(\Sigma ^{-1}\) is block diagonal and \(|\Sigma | = |\Sigma _{11}||\Sigma _{22}|\), so the density factorises into a function of \(x_1\) times a function of \(x_2\). □

Part (iii) is special to the normal distribution. For other distributions zero covariance does not imply independence (Problem 1.6.5 gave an example), but for jointly normal vectors it does, and that is what makes exact inference in the normal regression model possible.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.