2.5 Distribution of Quadratic Forms
Every sum of squares met so far — \(SSTr\), \(SSE\), \(SSB\), \(SST\) — is a quadratic form \(Y^tAY\) in the observation vector. The analysis of variance table asserts three things about them: that each has a chi-square distribution, that the numerator and denominator of the \(F\) ratio are independent, and that the degrees of freedom add up. None of the three has been proved. This subsection proves all three, and the idempotent matrices of Definition 2.3.5 are what does it: a sum of squares behaves like a chi-square exactly when its matrix is idempotent, and two sums of squares are independent exactly when their matrices multiply to zero.
Theorem 2.5.1 (Mean of a quadratic form). Let \(\underline {Y}\) be a random vector with \(E\left (\underline {Y}\right ) =\underline {\mu }\) and \(\text {Var}\left (\underline {Y}\right )=\Sigma \), and let \(A\) be a constant \(n\times n\) matrix. Then \[E\left (\underline {Y}^tA\underline {Y}\right ) = tr\left (A\Sigma \right ) + \underline {\mu }^tA\underline {\mu }.\]
Proof. Because \(\underline {Y}^tA\underline {Y}\) is a scalar it equals its own trace, and the trace is linear and invariant under cyclic permutation: \begin {align*} E\left (\underline {Y}^tA\underline {Y}\right ) & = E\left [tr\left (\underline {Y}^tA\underline {Y}\right )\right ] = E\left [tr\left (A\underline {Y}\,\underline {Y}^t\right )\right ]\\ & = tr\left [A\,E\left (\underline {Y}\,\underline {Y}^t\right )\right ] = tr\left [A\left (\Sigma +\underline {\mu }\,\underline {\mu }^t\right )\right ]\\ & = tr\left (A\Sigma \right )+tr\left (A\underline {\mu }\,\underline {\mu }^t\right ) = tr\left (A\Sigma \right )+\underline {\mu }^tA\underline {\mu }. \end {align*} □
Note. No normality is used, only the first two moments. This one identity produces every expected mean square in the course: put \(\Sigma =\sigma ^2I\) and it becomes \(E\left (\underline {Y}^tA\underline {Y}\right ) =\sigma ^2\,tr(A)+\underline {\mu }^tA\underline {\mu }\), so a quadratic form is an unbiased estimator of \(\sigma ^2\) precisely when \(tr(A)=1\) and \(A\underline {\mu }=\underline {0}\). That is the calculation which shows \(MSE\) estimates \(\sigma ^2\) whether or not the null hypothesis is true, while \(MSTr\) does so only under the null.
Definition 2.5.2 (Non-central chi-square). If \(\underline {Z}\sim N_n\left (\underline {\mu },I\right )\) then \(\underline {Z}^t\underline {Z}\) has the non-central chi-square distribution with \(n\) degrees of freedom and non-centrality parameter \(\lambda =\tfrac {1}{2}\underline {\mu }^t\underline {\mu }\), written \(\chi ^2\left (n,\lambda \right )\). When \(\lambda =0\) this is the ordinary chi-square.
Theorem 2.5.3 (Chi-square criterion). Let \(\underline {Y}\sim N_n\left (\underline {\mu },\sigma ^2I\right )\) and let \(A\) be symmetric. Then \[\frac {\underline {Y}^tA\underline {Y}}{\sigma ^2}\sim \chi ^2\left (r,\ \lambda \right ),\qquad r=rank(A),\quad \lambda =\frac {\underline {\mu }^tA\underline {\mu }}{2\sigma ^2},\] if and only if \(A\) is idempotent.
Proof. Take \(\sigma ^2=1\) without loss of generality. Since \(A\) is symmetric there is an orthogonal \(P\) with \(P^tAP=D=diag\left (\lambda _1,\dots ,\lambda _n\right )\). Put \(\underline {Z}=P^t\underline {Y}\), so that \(\underline {Z}\sim N_n\left (P^t\underline {\mu },I\right )\) and \[\underline {Y}^tA\underline {Y}=\underline {Z}^tD\underline {Z} =\sum _{i=1}^{n}\lambda _iZ_i^2 .\] The \(Z_i^2\) are independent non-central chi-squares on one degree of freedom. A weighted sum of independent chi-squares is itself chi-square only when every non-zero weight equals one, so the distribution is \(\chi ^2\left (r,\lambda \right )\) exactly when each \(\lambda _i\) is \(0\) or \(1\) with \(r\) of them equal to \(1\) — and a symmetric matrix has all its eigenvalues in \(\{0,1\}\) if and only if \(A^2=A\). For such an \(A\), \(rank(A)=tr(A)=r\). □
Note. This is the theorem the whole analysis of variance rests on, and it explains why idempotent matrices were introduced at all. It also explains the phrase “degrees of freedom”: for an idempotent matrix rank and trace coincide, so the degrees of freedom of a sum of squares can be read off as the trace of its matrix — a quantity that is usually obvious by inspection. For the residual sum of squares, \(A=I-H\) with \(H=X\left (X^tX\right )^{-1}X^t\), and \(tr(I-H)=n-tr(H)=n-k\).
Theorem 2.5.4 (Craig’s theorem). Let \(\underline {Y}\sim N_n\left (\underline {\mu },\sigma ^2I\right )\) and let \(A\) and \(B\) be symmetric. The quadratic forms \(\underline {Y}^tA\underline {Y}\) and \(\underline {Y}^tB\underline {Y}\) are independent if and only if \(AB=0\).
Theorem 2.5.5 (Independence of a linear and a quadratic form). With \(\underline {Y}\) as above and \(A\) symmetric, the linear form \(C\underline {Y}\) and the quadratic form \(\underline {Y}^tA\underline {Y}\) are independent if and only if \(CA=0\).
Note. Theorem 2.5.5 is why \(\overline {Y}\) and \(S^2\) are independent in the elementary one-sample theory, a fact usually quoted without proof. Take \(C=\tfrac {1}{n}\underline {1}^t\) and \(A=I-\tfrac {1}{n}J_n\); then \(CA=\tfrac {1}{n}\underline {1}^t-\tfrac {1}{n}\underline {1}^t=0\). The same argument, with \(C\) replaced by \(\left (X^tX\right )^{-1}X^t\) and \(A\) by \(I-H\), shows that \(\widehat {B}\) and \(SSE\) are independent in the general linear model, which is what allows a \(t\) statistic to be formed from them.
Theorem 2.5.6 (Cochran’s theorem). Let \(\underline {Y}\sim N_n\left (\underline {\mu },\sigma ^2I\right )\) and suppose \[\underline {Y}^t\underline {Y}=\underline {Y}^tA_1\underline {Y} +\underline {Y}^tA_2\underline {Y}+\cdots +\underline {Y}^tA_k\underline {Y},\] where each \(A_i\) is symmetric with \(rank(A_i)=r_i\). If \(r_1+r_2+\cdots +r_k=n\), then the \(\underline {Y}^tA_i\underline {Y}/\sigma ^2\) are mutually independent, each distributed as \(\chi ^2\left (r_i,\ \underline {\mu }^tA_i\underline {\mu }/2\sigma ^2\right )\).
Note. Cochran’s theorem is the economical form of the previous three. To justify an analysis of variance table one need not verify idempotency of each matrix and orthogonality of each pair separately: it is enough that the sums of squares add to the total and that the degrees of freedom add to \(n\). The arithmetic check at the foot of every analysis of variance table — that the degrees of freedom column sums correctly — is therefore not book-keeping. It is the hypothesis of a theorem.
Example 2.5.7. For the one-way layout of Section 1.2 with \(N\) observations in \(k\) groups, the identity \[SST = SSTr + SSE\] of that section is a decomposition of exactly this kind, with \[SST=\underline {Y}^t\left (I-\tfrac {1}{N}J_N\right )\underline {Y},\qquad rank=N-1,\] \[SSTr=\underline {Y}^t\left (\bigoplus _{j=1}^{k}\tfrac {1}{n_j}J_{n_j} -\tfrac {1}{N}J_N\right )\underline {Y},\qquad rank=k-1,\] \[SSE=\underline {Y}^t\left (I-\bigoplus _{j=1}^{k}\tfrac {1}{n_j}J_{n_j}\right ) \underline {Y},\qquad rank=N-k.\] Since \((k-1)+(N-k)=N-1\), Cochran’s theorem gives at once that \(SSTr/\sigma ^2\) and \(SSE/\sigma ^2\) are independent chi-square variates on \(k-1\) and \(N-k\) degrees of freedom. Under \(H_0:\tau _1=\cdots =\tau _k\) the non-centrality of the first vanishes, and \[F=\frac {SSTr/(k-1)}{SSE/(N-k)}\sim F_{k-1,\,N-k}\] by the definition of the \(F\) distribution as a ratio of independent chi-squares divided by their degrees of freedom.
Note. This example closes a gap left open in Section 1.2, where the \(F\) ratio was written down and its distribution asserted. The assertion is now a consequence. Notice also what happens when \(H_0\) is false: only the non-centrality of \(SSTr\) changes, \(SSE\) being unaffected because \(A_{SSE}\underline {\mu }=\underline {0}\) whatever the treatment means are. The \(F\) statistic is therefore stochastically larger under the alternative, which is why the test rejects in the upper tail only.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.