3.3 The analysis of variance

Write \(\bar y_{i\cdot } = y_{i\cdot }/n_i\) and \(\bar y_{\cdot \cdot } = y_{\cdot \cdot }/N\). The deviation of each observation from the grand mean splits into a between-treatment and a within-treatment part, \[ y_{ij} - \bar y_{\cdot \cdot } = (\bar y_{i\cdot } - \bar y_{\cdot \cdot }) + (y_{ij} - \bar y_{i\cdot }) . \]

Theorem 3.3.1 (Partition of the total sum of squares). \[ \underbrace {\sum _{i=1}^t\sum _{j=1}^{n_i}(y_{ij} - \bar y_{\cdot \cdot })^2}_{SST} = \underbrace {\sum _{i=1}^tn_i(\bar y_{i\cdot } - \bar y_{\cdot \cdot })^2}_{SS_{\mathrm {Treatments}}} + \underbrace {\sum _{i=1}^t\sum _{j=1}^{n_i}(y_{ij} - \bar y_{i\cdot })^2}_{SSE} . \]

Proof. Square the decomposition above and sum over \(i\) and \(j\). The cross-product term is \[ 2\sum _{i=1}^t(\bar y_{i\cdot } - \bar y_{\cdot \cdot })\sum _{j=1}^{n_i}(y_{ij} - \bar y_{i\cdot }) = 0 , \] because the deviations within each treatment sum to zero. □

The degrees of freedom split in the same way: \(N - 1 = (t - 1) + (N - t)\). For calculation by hand, \begin {gather*} SST = \sum _i\sum _jy_{ij}^2 - \frac {y_{\cdot \cdot }^2}{N} , \qquad SS_{\mathrm {Treatments}} = \sum _i\frac {y_{i\cdot }^2}{n_i} - \frac {y_{\cdot \cdot }^2}{N} , \\ SSE = SST - SS_{\mathrm {Treatments}} , \end {gather*} where \(y_{\cdot \cdot }^2/N\) is called the correction factor. The mean squares are the sums of squares divided by their degrees of freedom, \[ MS_{\mathrm {Treatments}} = \frac {SS_{\mathrm {Treatments}}}{t - 1} , \qquad MSE = \frac {SSE}{N - t} . \]

Theorem 3.3.2 (Expected mean squares). Under the effects model with \(E(\varepsilon _{ij}) = 0\), \(\Var (\varepsilon _{ij}) = \sigma ^2\) and uncorrelated errors (normality is not needed), \[ E(MSE) = \sigma ^2 , \qquad E(MS_{\mathrm {Treatments}}) = \sigma ^2 + \frac {\sum _{i=1}^tn_i(\tau _i - \bar \tau )^2}{t - 1} , \] where \(\bar \tau = \sum n_i\tau _i/N\). In a balanced experiment, \(n_i = r\) for all \(i\), this is \[ E(MS_{\mathrm {Treatments}}) = \sigma ^2 + \frac {r}{t - 1}\sum _{i=1}^t(\tau _i - \bar \tau )^2 , \qquad \bar \tau = \frac 1t\sum _{i=1}^t\tau _i . \]

Proof. We use \(E(W^2) = \Var (W) + \bigl (E(W)\bigr )^2\) repeatedly. Write \(\mu _i = \mu + \tau _i\) and \(\bar \mu = \sum n_i\mu _i/N = \mu + \bar \tau \). Then \begin {align*} E(y_{ij}^2) &= \sigma ^2 + \mu _i^2 , \\ E(y_{i\cdot }^2) &= \Var (y_{i\cdot }) + \bigl (E(y_{i\cdot })\bigr )^2 = n_i\sigma ^2 + n_i^2\mu _i^2 , \\ E(y_{\cdot \cdot }^2) &= N\sigma ^2 + N^2\bar \mu ^2 . \end {align*}

For the error sum of squares, \begin {align*} E(SSE) &= \sum _i\sum _jE(y_{ij}^2) - \sum _i\frac {E(y_{i\cdot }^2)}{n_i} \\ &= \sum _in_i(\sigma ^2 + \mu _i^2) - \sum _i(\sigma ^2 + n_i\mu _i^2) = (N - t)\sigma ^2 . \end {align*}

For the treatment sum of squares, \begin {align*} E(SS_{\mathrm {Treatments}}) &= \sum _i\frac {E(y_{i\cdot }^2)}{n_i} - \frac {E(y_{\cdot \cdot }^2)}{N} \\ &= \sum _i(\sigma ^2 + n_i\mu _i^2) - (\sigma ^2 + N\bar \mu ^2) \\ &= (t - 1)\sigma ^2 + \sum _in_i\mu _i^2 - N\bar \mu ^2 \\ &= (t - 1)\sigma ^2 + \sum _in_i(\mu _i - \bar \mu )^2 , \end {align*}

and \(\mu _i - \bar \mu = \tau _i - \bar \tau \). Dividing by the degrees of freedom gives both results, and with \(n_i = r\), \(\bar \tau \) is the simple average of the \(\tau _i\). □

With the side condition \(\sum n_i\tau _i = 0\) we have \(\bar \tau = 0\), and the result is often quoted as \[ E(MS_{\mathrm {Treatments}}) = \sigma ^2 + \frac {\sum n_i\tau _i^2}{t - 1} . \] Either way, \(MSE\) estimates \(\sigma ^2\) whatever the treatment effects, while \(MS_{\mathrm {Treatments}}\) estimates \(\sigma ^2\) only when all the \(\tau _i\) are equal and something larger otherwise. This suggests comparing the two.

Theorem 3.3.3. Under the effects model with normal errors,

(i): \(SSE/\sigma ^2 \sim \chi ^2_{N-t}\);
(ii): \(SS_{\mathrm {Treatments}}/\sigma ^2 \sim \chi ^2_{t-1}(\lambda )\) with \(\lambda = \sum _in_i(\tau _i - \bar \tau )^2/\sigma ^2\);
(iii): \(SSE\) and \(SS_{\mathrm {Treatments}}\) are independent.

Hence \(F_0 = MS_{\mathrm {Treatments}}/MSE \sim F_{t-1,\,N-t}(\lambda )\), and under \(H_0 : \tau _1 = \dots = \tau _t\), where \(\lambda = 0\), \(F_0 \sim F_{t-1,\,N-t}\).

Proof. Let \(H\) be the projection onto the column space of the design matrix, which is spanned by the \(t\) treatment indicators; \(H\mathbf Y\) replaces each observation by its treatment mean \(\bar y_{i\cdot }\). Let \(H_0 = \frac 1NJ\), the projection onto the column of ones, which replaces each observation by \(\bar y_{\cdot \cdot }\). Since the column of ones lies in \(\mathcal C(X)\), \(HH_0 = H_0H = H_0\). Then \[ SSE = \mathbf Y^\top (I - H)\mathbf Y , \qquad SS_{\mathrm {Treatments}} = \mathbf Y^\top (H - H_0)\mathbf Y , \] because \((H - H_0)\mathbf Y\) has entries \(\bar y_{i\cdot } - \bar y_{\cdot \cdot }\). Both \(I - H\) and \(H - H_0\) are symmetric and idempotent, for example \((H - H_0)^2 = H - H_0 - H_0 + H_0 = H - H_0\), with ranks equal to their traces, \(N - t\) and \(t - 1\); and \((I - H)(H - H_0) = H - H_0 - H + H_0 = 0\). By Theorem 3.1.4 the two sums of squares are independent with the stated distributions. The non-centrality parameters are found by replacing \(\mathbf Y\) by its mean: \((I - H)\boldsymbol \mu = \mathbf 0\), while \((H - H_0)\boldsymbol \mu \) has entries \(\mu _i - \bar \mu = \tau _i - \bar \tau \), so \(\boldsymbol \mu ^\top (H - H_0)\boldsymbol \mu = \sum n_i(\tau _i - \bar \tau )^2\). □

We therefore test \(H_0 : \tau _1 = \tau _2 = \dots = \tau _t\) against \(H_1\): \(\tau _i \ne \tau _k\) for at least one pair, by rejecting \(H_0\) when \[ F_0 = \frac {MS_{\mathrm {Treatments}}}{MSE} > F_{\alpha ,\,t-1,\,N-t} . \] The test is one-sided because, by Theorem 3.3.2, departures from \(H_0\) can only make \(F_0\) large. The calculations are set out in an analysis of variance (ANOVA) table:

source	sum of squares	df	mean square	\(F_0\)

treatments	\(SS_{\mathrm {Treatments}}\)	\(t - 1\)	\(MS_{\mathrm {Treatments}}\)	\(F_0\)
error	\(SSE\)	\(N - t\)	\(MSE\)

total	\(SST\)	\(N - 1\)

Remark. The \(F\) test is a comparison of two models fitted by least squares. The full model \(y_{ij} = \mu + \tau _i + \varepsilon _{ij}\) has residual sum of squares \(SSE\). The reduced model \(y_{ij} = \mu + \varepsilon _{ij}\), which is the model under \(H_0\), has residual sum of squares \(SST\). The treatment sum of squares is the reduction in residual sum of squares achieved by adding the treatment effects, \(SS_{\mathrm {Treatments}} = SST - SSE\), and \(F_0\) compares that reduction, per parameter added, with the error mean square of the full model. This extra sum of squares principle is how every \(F\) test in this course, and in regression, is constructed. It is also the likelihood ratio test of \(H_0\).

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.