9.3 Factors with more than two levels

A factor with \(a\) levels needs \(a - 1\) indicators. With three levels \(A\), \(B\), \(C\) and reference level \(A\), let \(z_1 = 1\) for level \(B\) and \(z_2 = 1\) for level \(C\), both \(0\) otherwise:

Level \(z_1\) \(z_2\)
\(A\) 0 0
\(B\) 1 0
\(C\) 0 1

In \(y = \beta _0 + \beta _1x + \beta _2z_1 + \beta _3z_2 + \varepsilon \), the three groups have parallel lines with intercepts \(\beta _0\), \(\beta _0 + \beta _2\) and \(\beta _0 + \beta _3\), so \(\beta _2\) and \(\beta _3\) compare \(B\) and \(C\) with \(A\); \(B\) is compared with \(C\) through \(\beta _2 - \beta _3\). Whether the factor matters at all is the test of \(H_0\colon \beta _2 = \beta _3 = 0\), a partial \(F\) on \(2\) degrees of freedom; testing \(\beta _2\) and \(\beta _3\) separately does not answer that question.

It would be wrong to code the three levels as one regressor taking the values \(1\), \(2\), \(3\). That forces the effect of moving from \(A\) to \(B\) to equal the effect of moving from \(B\) to \(C\), an assumption with no basis for categories that have no order.

Theorem 9.3.1 (Regression and one-way analysis of variance). Suppose the only regressors are the \(a - 1\) indicators of a factor, with \(n_i\) observations at level \(i\). Then

(i)
the fitted value of every observation is the mean \(\bar y_i\) of its level; \(\hat \beta _0\) is the reference-level mean, and the coefficient of each indicator is the difference between its level’s mean and the reference-level mean;
(ii)
\(SS_R = \sum _i n_i(\bar y_i - \bar y)^2\) and \(SS_{\mathrm {Res}} = \sum _i\sum _j(y_{ij} - \bar y_i)^2\), the between-groups and within-groups sums of squares, and the \(F\)-test for significance of regression is the one-way analysis of variance \(F\)-test of equal level means.

Proof. (i) The intercept and the \(a - 1\) indicators span the same space as the \(a\) indicators of all the levels (the reference indicator is the ones column minus the others). That space consists of the vectors that are constant within each level, and, as in the proof of Theorem 4.6.2, projecting onto it replaces each \(y_{ij}\) by its level mean. Matching \(\beta _0 + \beta _j\) with \(\bar y_j\) and \(\beta _0\) with the reference mean gives the coefficients.

(ii) \(SS_{\mathrm {Res}} = \sum (y_{ij} - \hat y_{ij})^2\) with \(\hat y_{ij} = \bar y_i\), and \(SS_R = \sum (\hat y_{ij} - \bar y)^2 = \sum _i n_i(\bar y_i - \bar y)^2\). The regression has \(a - 1\) degrees of freedom and the residual \(n - a\), the same as the analysis of variance, so the two \(F\) statistics coincide. □

So the one-way analysis of variance is a special case of regression, and Theorem 6.4.3 and Theorem 4.6.1 were already using the same structure. Adding a quantitative regressor \(x\) to the indicators gives the analysis of covariance.

Example 9.3.2. Twelve learners were taught a topic by one of three methods, four by each, and their test scores were:

Method \(A\) 5 7 6 6
Method \(B\) 9 8 10 9
Method \(C\) 7 8 6 7

Fit a regression with indicators for \(B\) and \(C\), and test whether the method matters.

Solution. The level means are \(\bar y_A = 6\), \(\bar y_B = 9\), \(\bar y_C = 7\) and \(\bar y = 88/12 = 7.333\). By Theorem 9.3.1, \[ \hat y = 6 + 3z_1 + 1z_2 . \] Method \(B\) scores \(3\) marks more than method \(A\) on average, method \(C\) \(1\) mark more. Then \begin {align*} SS_R &= 4\bigl [(6 - 7.333)^2 + (9 - 7.333)^2 + (7 - 7.333)^2\bigr ] \\ &= 4(1.778 + 2.778 + 0.111) = 18.67 , \\ SS_{\mathrm {Res}} &= (1 + 1 + 0 + 0) + (0 + 1 + 1 + 0) + (0 + 1 + 1 + 0) = 6 , \end {align*}

on \(2\) and \(9\) degrees of freedom, so \[ F_0 = \frac {18.67/2}{6/9} = \frac {9.333}{0.667} = 14.0 > F_{0.05,2,9} = 4.26 . \] The method matters. The same table is the one-way analysis of variance of the three groups. With \(MS_{\mathrm {Res}} = 0.667\) and four learners per group, each comparison with \(A\) has standard error \(\sqrt {0.667(1/4 + 1/4)} = 0.577\), so \(\hat \beta _1 = 3\) gives \(t = 5.2\) and \(\hat \beta _2 = 1\) gives \(t = 1.73\): \(B\) is clearly better than \(A\), while the evidence for \(C\) over \(A\) is weak.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.