4.4 Multiple Comparisons
A significant Friedman test says that the \(k\) treatments are not all alike. It does not say which differ from which, and answering that requires a post-hoc procedure.
The Nemenyi test is the rank analogue of Tukey’s test, and is used when every treatment is compared with every other. Two treatments \(i\) and \(j\) are declared different when their mean ranks differ by at least the critical difference \[CD=q_{\alpha }\sqrt {\dfrac {k(k+1)}{6n}}, \hspace {0.8cm}\text {that is, when}\hspace {0.4cm} \left |\overline {R}_{\cdot i}-\overline {R}_{\cdot j}\right |\geq CD ,\] where \(\overline {R}_{\cdot j}=R_{\cdot j}/n\) is the mean rank of treatment \(j\) over the \(n\) blocks. The value \(q_{\alpha }\) comes from the studentised range distribution, \(q_{\alpha }=q_{\infty ,k,\alpha }/\sqrt {2}\).
Note that \(CD\) is a threshold the observed difference is compared against, not the observed difference itself. Comparing mean ranks with mean ranks matters too: the standard error above is that of \(\overline {R}_{\cdot i}-\overline {R}_{\cdot j}\), so putting rank sums in the numerator inflates every comparison by a factor of \(n\).
When all treatments are compared against a single control rather than against each other, there are only \(k-1\) comparisons instead of \(\binom {k}{2}\), and a general family-wise procedure such as Bonferroni can be used instead. The pairwise statistic is \[Z=\dfrac {\overline {R}_{\cdot i}-\overline {R}_{\cdot j}}{\sqrt {\dfrac {k(k+1)}{6n}}} \ \thicksim \ N(0,1),\] referred to the normal table and compared with an adjusted level. Bonferroni divides \(\alpha \) by the number of comparisons made, \(k-1\) in the control case; the same test can be carried out as a critical difference by using \(q\) at level \(\alpha /k\). These adjustments are conservative in general, but with only \(k-1\) comparisons to protect they are more powerful than Nemenyi, which pays for all \(\binom {k}{2}\).
Example 4.13. Six raters each score four restaurants for service. The table gives the ratings and the ranks assigned within each block.
| A | B | C | D | |||||
| Blocks of | Rating | Rank | Rating | Rank | Rating | Rank | Rating | Rank |
| Raters | ||||||||
| 1 | 70 | 2 | 61 | 1 | 82 | 4 | 74 | 3 |
| 2 | 77 | 3 | 75 | 1 | 88 | 4 | 76 | 2 |
| 3 | 76 | 2 | 67 | 1 | 90 | 4 | 80 | 3 |
| 4 | 80 | 3 | 63 | 1 | 96 | 4 | 76 | 2 |
| 5 | 84 | 2.5 | 66 | 1 | 92 | 4 | 84 | 2.5 |
| 6 | 78 | 2 | 68 | 1 | 98 | 4 | 86 | 3 |
| \(R_{\cdot 1}\) | \(=14.5\) | \(R_{\cdot 2}\) | \(=6\) | \(R_{\cdot 3}\) | \(=24\) | \(R_{\cdot 4}\) | \(=15.5\) | |
Test at \(\alpha =0.05\) whether the four restaurants are rated alike.
Solution. Here \(k=4\) treatments and \(n=6\) blocks. Ranking is done within each rater, so rater 5, who gave \(84\) to both A and D, splits ranks \(2\) and \(3\) between them as mid-ranks of \(2.5\).
As a check the rank sums must total \(nk(k+1)/2=6\times 4\times 5/2=60\), and indeed \(14.5+6+24+15.5=60\).
\[F_R=\dfrac {12}{nk(k+1)}\sum ^{k}_{j=1}R^{2}_{\cdot j}-3n(k+1)\] \begin {align*} F_R &=\dfrac {12}{6\times 4\times 5}\left (14.5^{2}+6^{2}+24^{2}+15.5^{2}\right )-3\times 6\times 5\\ &=\dfrac {12}{120}\left (210.25+36+576+240.25\right )-90\\ &=0.1(1062.5)-90\\ &=106.25-90=16.25 . \end {align*}
Tie correction. There is one group of tied ranks, of size \(t=2\), in block 5. Dividing by \[1-\dfrac {\sum \left (t_p^{3}-t_p\right )}{n\left (k^{3}-k\right )} =1-\dfrac {2^{3}-2}{6\left (4^{3}-4\right )}=1-\dfrac {6}{360}=0.9833\] gives the corrected value \(F_R=16.25/0.9833=16.53\). With a single tie the difference is negligible, as it usually is.
Decision. With \(k-1=3\) degrees of freedom, \(\chi ^{2}_{(0.05,3)}=7.815\). Since \[F_R=16.25>7.815\] we reject \(H_0\) at the \(5\%\) level: the four restaurants are not rated alike. (The exact critical value for \(k=4\), \(n=6\) is \(7.600\), which the design is small enough to warrant; the conclusion is the same either way.)
Which differ? The mean ranks \(\overline {R}_{\cdot j}=R_{\cdot j}/n\) are \[\overline {R}_{A}=2.417,\hspace {0.5cm}\overline {R}_{B}=1.000, \hspace {0.5cm}\overline {R}_{C}=4.000,\hspace {0.5cm}\overline {R}_{D}=2.583 .\] With \(q_{0.05}=q_{\infty ,4,0.05}/\sqrt {2}=3.633/\sqrt {2}=2.569\), \[CD=2.569\sqrt {\dfrac {k(k+1)}{6n}}=2.569\sqrt {\dfrac {4\times 5}{6\times 6}} =2.569\times 0.745=1.915 .\] Comparing all \(\binom {4}{2}=6\) pairs against this threshold:
| pair | A–B | A–C | A–D | B–C | B–D | C–D |
| \(\left |\overline {R}_{\cdot i}-\overline {R}_{\cdot j}\right |\) | 1.417 | 1.583 | 0.167 | 3.000 | 1.583 | 1.417 |
| \(\geq CD=1.915\)? | no | no | no | yes | no | no |
Only C against B is separated: the highest-rated restaurant differs from the lowest, and no other pair can be distinguished. This is the usual outcome of a post-hoc test on a small design, and it is the honest one — the omnibus test found some difference, and six raters are enough to locate only the largest of them.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.