4.3 Multiple comparisons
A \(95\%\) confidence interval misses its target \(5\%\) of the time. Make twenty such intervals and, even if each is valid, about one of them is expected to be wrong. The more comparisons an experimenter makes, and especially the more he or she chooses them by looking at the data, the more likely it is that some apparent difference is spurious.
Definition 4.3.1. For a family of tests, the comparisonwise error rate is the probability of a type I error on any one test, and the familywise error rate is the probability of at least one type I error in the family. A procedure controls the familywise error rate at \(\alpha \) if this probability is at most \(\alpha \) whatever the true means. The corresponding property for intervals is simultaneous coverage: the probability that every interval in the family contains its parameter is at least \(1 - \alpha \).
The methods below are for a balanced design with \(r\) replicates per treatment, the case of most practical interest; each extends to unequal \(n_i\).
Fisher’s least significant difference
Two means are declared different if \[ |\bar y_{i\cdot } - \bar y_{k\cdot }| > LSD = t_{\alpha /2,\,N-t}\sqrt {\frac {2\,MSE}{r}} . \] This controls only the comparisonwise rate. In the protected form it is used only after the \(F\) test has rejected \(H_0\); that controls the familywise rate when all the means are equal, but not when some are equal and others are not.
Bonferroni’s method
Proposition 4.3.2. If each of \(m\) tests is carried out at level \(\alpha /m\), the familywise error rate is at most \(\alpha \). Equivalently, \(m\) intervals each with confidence \(1 - \alpha /m\) have simultaneous coverage at least \(1 - \alpha \).
Proof. Let \(E_k\) be the event that the \(k\)th interval misses. By Boole’s inequality, \(P(E_1 \cup \dots \cup E_m) \le \sum _kP(E_k) = m(\alpha /m) = \alpha \). □
The method needs no assumption about the dependence between the tests, and works for any pre-specified family of \(m\) contrasts, using \(t_{\alpha /(2m),\,N-t}\) in place of \(t_{\alpha /2,\,N-t}\). It becomes conservative when \(m\) is large.
Tukey’s method
For all \(\binom t2\) pairwise differences, Tukey’s method is exact.
Definition 4.3.3. If \(Z_1, \dots , Z_t\) are independent \(N(0, 1)\) and \(V \sim \chi ^2_\nu \) is independent of them, the distribution of \[ Q = \frac {\max _iZ_i - \min _iZ_i}{\sqrt {V/\nu }} \] is the studentised range distribution with parameters \(t\) and \(\nu \). Its upper \(\alpha \) point is written \(q_\alpha (t, \nu )\).
Theorem 4.3.4 (Tukey). In a balanced one-way design, with probability exactly \(1 - \alpha \) all the intervals \[ \bar y_{i\cdot } - \bar y_{k\cdot } \pm q_\alpha (t, N - t)\sqrt {\frac {MSE}{r}} , \qquad 1 \le i < k \le t , \] contain the corresponding \(\mu _i - \mu _k\) simultaneously.
Proof. Let \(Z_i = (\bar y_{i\cdot } - \mu _i)/(\sigma /\sqrt r)\), which are independent \(N(0, 1)\), and \(V = (N - t)MSE/\sigma ^2 \sim \chi ^2_{N-t}\), independent of them. Every interval covers if and only if \(|(\bar y_{i\cdot } - \mu _i) - (\bar y_{k\cdot } - \mu _k)| \le q_\alpha \sqrt {MSE/r}\) for all pairs, that is, if and only if the largest such difference, which is \(\max _i(\bar y_{i\cdot } - \mu _i) - \min _i(\bar y_{i\cdot } - \mu _i)\), is at most \(q_\alpha \sqrt {MSE/r}\). Dividing by \(\sqrt {MSE/r}\), this is the event \((\max Z_i - \min Z_i)/\sqrt {V/(N - t)} \le q_\alpha \), which has probability \(1 - \alpha \) by definition. □
The quantity \(HSD = q_\alpha (t, N - t)\sqrt {MSE/r}\) is Tukey’s honestly significant difference. For unequal \(n_i\), replacing \(\sqrt {MSE/r}\) by \(\sqrt {MSE(1/n_i + 1/n_k)/2}\) gives the Tukey–Kramer method, which is slightly conservative.
Scheffé’s method
Scheffé’s method covers every contrast, including those suggested by the data.
Theorem 4.3.5 (Scheffé). Let \(S^2 = (t - 1)F_{\alpha ,\,t-1,\,N-t}\). With probability exactly \(1 - \alpha \), the intervals \[ \hat C \pm S\sqrt {MSE\sum _{i=1}^t\frac {c_i^2}{n_i}} \] contain \(C\) simultaneously for all contrasts \(C = \sum c_i\mu _i\).
Proof. Let \(u_i = \bar y_{i\cdot } - \mu _i\) and \(\bar u = \sum n_iu_i/N\). For any contrast, since \(\sum c_i = 0\), \[ \hat C - C = \sum _ic_iu_i = \sum _ic_i(u_i - \bar u) = \sum _i\frac {c_i}{\sqrt {n_i}}\cdot \sqrt {n_i}(u_i - \bar u) . \] By the Cauchy–Schwarz inequality, \[ (\hat C - C)^2 \le \Bigl (\sum _i\frac {c_i^2}{n_i}\Bigr )\sum _in_i(u_i - \bar u)^2 , \] with equality when \(c_i \propto n_i(u_i - \bar u)\), which is itself a contrast. Hence \[ \max _{\text {contrasts}}\frac {(\hat C - C)^2}{MSE\sum c_i^2/n_i} = \frac {\sum _in_i(u_i - \bar u)^2}{MSE} . \] The numerator is the treatment sum of squares computed from the errors \(\varepsilon _{ij}\) in place of the \(y_{ij}\), so it is \(\sigma ^2\chi ^2_{t-1}\), independent of \(MSE\), and the right side is distributed as \((t - 1)F_{t-1,\,N-t}\). So every interval covers if and only if this maximum is at most \(S^2\), which has probability \(1 - \alpha \). □
The price of covering infinitely many contrasts is wide intervals. Scheffé’s method is the right one when contrasts are chosen after looking at the data; for pairwise comparisons alone, Tukey’s is better. The two are linked: the \(F\) test rejects \(H_0\) if and only if at least one contrast is significant by Scheffé’s criterion.
Example 4.3.6. For the coating data, \(r = 4\), \(MSE = 8.15\) and \(N - t = 15\). For pairwise differences, the minimum significant difference is the multiplier times \(\sqrt {2(8.15)/4} = 2.019\), except for Tukey’s method, where it is the multiplier times \(\sqrt {8.15/4} = 1.427\):
| method | multiplier | minimum difference |
| Fisher LSD | \(t_{0.025,15} = 2.131\) | \(4.30\) |
| Tukey | \(q_{0.05}(5, 15) = 4.367\) | \(6.23\) |
| Bonferroni, \(m = 10\) | \(t_{0.0025,15} = 3.286\) | \(6.63\) |
| Scheffé | \(\sqrt {4F_{0.05,4,15}} = 3.496\) | \(7.06\) |
The means in increasing order are \(62.75\) (coating \(5\)), \(65.00\) (\(4\)), \(71.75\) (\(3\)), \(77.00\) (\(2\)) and \(82.75\) (\(1\)). By Tukey’s method, the pairs not significantly different are \(\{4, 5\}\) (difference \(2.25\)), \(\{2, 3\}\) (\(5.25\)) and \(\{1, 2\}\) (\(5.75\)); all other pairs differ. A compact way to show this gives each mean one or more letters, so that two means share a letter exactly when they are not significantly different:
| coating | \(5\) | \(4\) | \(3\) | \(2\) | \(1\) |
| mean | \(62.75\) | \(65.00\) | \(71.75\) | \(77.00\) | \(82.75\) |
| group | a | a | b | bc | c |
Bonferroni gives the same groups. The LSD separates every pair except \(\{4, 5\}\), which overstates the evidence. Scheffé’s method, the most conservative, also fails to separate coatings \(3\) and \(4\) (difference \(6.75 < 7.06\)). The different answers for that one pair are a reminder that a difference close to the borderline is uncertain, whichever method is used.
Remark. When one treatment is a control and the only questions are whether each other treatment differs from it, Dunnett’s method uses the joint distribution of the \(t - 1\) comparisons with the control and gives shorter intervals than Tukey’s. In every case the family of comparisons should be decided when the experiment is designed. Planned contrasts that answer the questions the experiment was built for need no correction, or only Bonferroni’s; it is the unplanned search through the data that requires Tukey or Scheffé.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.