5.4 Distributions of quadratic forms

Definition 5.4.1. If \(Z_1, \dots , Z_r\) are independent \(N(0, 1)\), then \(\sum Z_i^2\) has the chi-square distribution with \(r\) degrees of freedom, \(\chi ^2_r\). It has mean \(r\), variance \(2r\), and moment generating function \((1 - 2t)^{-r/2}\) for \(t < \frac 12\). If \(W_1 \sim \chi ^2_{r_1}\) and \(W_2 \sim \chi ^2_{r_2}\) are independent, \(\dfrac {W_1/r_1}{W_2/r_2} \sim F_{r_1, r_2}\); and if \(Z \sim N(0, 1)\) is independent of \(W \sim \chi ^2_r\), \(Z/\sqrt {W/r} \sim t_r\).

Theorem 5.4.2.

(i): If \(X \sim N(0, \sigma ^2I_k)\), then \(X'(\sigma ^2I)^{-1}X = X'X/\sigma ^2 \sim \chi ^2_k\).
(ii): If \(X \sim N(0, \Sigma )\) with \(\Sigma \) an \(n \times n\) pd matrix, then \(X'\Sigma ^{-1}X \sim \chi ^2_n\).

Proof. (i) The elements \(X_i/\sigma \) are independent \(N(0, 1)\), and \(X'X/\sigma ^2 = \sum (X_i/\sigma )^2\).

(ii) Let \(Z = \Sigma ^{-1/2}X\). By Theorem 5.3.2(ii) and Corollary 5.1.4(iii), \(Z \sim N(0, \Sigma ^{-1/2}\Sigma \Sigma ^{-1/2}) = N(0, I_n)\). Since \(\Sigma ^{-1} = \Sigma ^{-1/2}\Sigma ^{-1/2}\) with \(\Sigma ^{-1/2}\) symmetric, \(X'\Sigma ^{-1}X = (\Sigma ^{-1/2}X)'(\Sigma ^{-1/2}X) = Z'Z \sim \chi ^2_n\) by (i). □

Part (ii) is a multivariate standardisation: subtracting the mean and dividing by the standard deviation turns a normal variable into a \(N(0, 1)\), and premultiplying by \(\Sigma ^{-1/2}\) turns a normal vector into a vector of independent \(N(0, 1)\)’s. The quadratic form \((x - \mu )'\Sigma ^{-1}(x - \mu )\) is the squared Mahalanobis distance of \(x\) from \(\mu \), the multivariate counterpart of a squared \(z\)-score. Every Wald statistic in econometrics has this shape.

Theorem 5.4.3. If \(X \sim N(0, I_n)\) and \(A\) is an \(n \times n\) symmetric idempotent matrix of rank \(r\), then \(X'AX \sim \chi ^2_r\).

Proof. By Theorems 5.1.3 and 5.1.5, \(A = Q\Lambda Q'\) with \(Q\) orthogonal and \(\Lambda \) diagonal with \(r\) ones and \(n - r\) zeros; order the columns of \(Q\) so the ones come first. Let \(Y = Q'X\). Then \(Y \sim N(0, Q'IQ) = N(0, I_n)\), and \[ X'AX = X'Q\Lambda Q'X = Y'\Lambda Y = Y_1^2 + \dots + Y_r^2 \sim \chi ^2_r . \qedhere \] □

The converse is also true: if \(A\) is symmetric and \(X'AX \sim \chi ^2_r\) for \(X \sim N(0, I)\), then \(A\) is idempotent of rank \(r\). It is proved by comparing the moment generating function \(\prod _i(1 - 2t\lambda _i)^{-1/2}\) of \(X'AX\), obtained from the same diagonalisation, with \((1 - 2t)^{-r/2}\).

Independence

Regression statistics are ratios, and the \(t\) and \(F\) distributions require the numerator and denominator to be independent. The next theorem gives simple sufficient conditions.

Theorem 5.4.4. Let \(X \sim N(0, \sigma ^2I_n)\), let \(A\) and \(B\) be symmetric idempotent, and let \(L\) be any \(m \times n\) matrix.

(i): If \(LA = 0\), the linear form \(LX\) and the quadratic form \(X'AX\) are independent.
(ii): If \(AB = 0\), the quadratic forms \(X'AX\) and \(X'BX\) are independent.

Proof. (i) Stack \(LX\) and \(AX\) into one vector, which is normal by Theorem 5.3.2(ii), applied to a suitable full-rank selection of its elements. Their cross-covariance is \(\Cov (LX, AX) = L(\sigma ^2I)A' = \sigma ^2LA = 0\), so \(LX\) and \(AX\) are independent by Theorem 5.3.2(iii). But \(X'AX = X'A'AX = (AX)'(AX)\) is a function of \(AX\) alone, so it is independent of \(LX\).

(ii) \(\Cov (AX, BX) = \sigma ^2AB' = \sigma ^2AB = 0\), so \(AX\) and \(BX\) are independent as in (i), and \(X'AX = (AX)'(AX)\) and \(X'BX = (BX)'(BX)\) are functions of them. □

Example 5.4.5. For a random sample from \(N(\mu , \sigma ^2)\), \(\bar X\) and \(S^2 = \frac {1}{n-1}\sum (X_i - \bar X)^2\) are independent and \((n - 1)S^2/\sigma ^2 \sim \chi ^2_{n-1}\).

Let \(X = \mu \iota + U\) with \(U \sim N(0, \sigma ^2I)\). Then \(\bar X = \mu + \frac 1n\iota 'U\) and \((n - 1)S^2 = X'M_0X = U'M_0U\), because \(M_0\iota = 0\). With \(L = \frac 1n\iota '\), \(LM_0 = \frac 1n(M_0\iota )' = 0\), so \(\bar X\) and \(S^2\) are independent by Theorem 5.4.4(i). And \(U/\sigma \sim N(0, I)\) with \(M_0\) symmetric idempotent of rank \(n - 1\) (Example 5.1.6), so \(U'M_0U/\sigma ^2 \sim \chi ^2_{n-1}\) by Theorem 5.4.3. Together these give \(\sqrt n(\bar X - \mu )/S \sim t_{n-1}\), the one-sample \(t\) test.

Theorem 5.4.6 (Cochran). Let \(X \sim N(0, I_n)\) and let \(A_1, \dots , A_m\) be symmetric matrices with \(A_1 + \dots + A_m = I_n\) and ranks \(r_1, \dots , r_m\). If \(r_1 + \dots + r_m = n\), then the quadratic forms \(X'A_iX\) are independent with \(X'A_iX \sim \chi ^2_{r_i}\).

Proof. By Theorem 5.1.3 each \(A_i\) has \(r_i\) non-zero eigenvalues \(\lambda _{ij}\) with orthonormal eigenvectors \(q_{ij}\), and \(A_i = \sum _{j=1}^{r_i}\lambda _{ij}q_{ij}q_{ij}'\). Collect all the eigenvectors, \(n\) of them in total, as the columns of an \(n \times n\) matrix \(P\), and the eigenvalues in the diagonal matrix \(D\) in the same order. Then \[ I_n = \sum _iA_i = PDP' . \] So \(P\) has rank \(n\) and is invertible. Multiplying on the left by \(P'\) and on the right by \(P\) gives \((P'P)D(P'P) = P'P\), and cancelling the invertible \(P'P\) gives \(D(P'P) = I\), that is \(P'P = D^{-1}\). The left side has diagonal elements \(q_{ij}'q_{ij} = 1\), so \(D^{-1}\), and hence \(D\), is the identity: every \(\lambda _{ij} = 1\), and \(P'P = I\), so \(P\) is orthogonal. Now let \(Y = P'X \sim N(0, I_n)\). Each \(A_i = \sum _jq_{ij}q_{ij}'\), so \[ X'A_iX = \sum _{j=1}^{r_i}(q_{ij}'X)^2 \] is the sum of squares of the \(r_i\) elements of \(Y\) that belong to block \(i\). The blocks use disjoint sets of the independent \(N(0, 1)\) elements of \(Y\), so the quadratic forms are independent \(\chi ^2_{r_i}\) variables. □

The proof shows more than the statement: when the ranks add up to \(n\), every \(A_i\) is automatically idempotent and the \(A_i\) are mutually orthogonal. Cochran’s theorem is the formal basis of every analysis of variance. A total sum of squares is split into components whose matrices add to \(I\) and whose degrees of freedom add to \(n\), and the theorem then guarantees that the components are independent \(\chi ^2\)’s, so that their mean squares can be compared by \(F\) tests.

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.