3.5 Tests Comparing Two Populations

Everything so far has tested a single population against a stated value. Most practical questions are comparisons instead: is this method better than that one, does this brand outlast the other, do men and women differ. The machinery is the same — the four steps do not change — and only the standard error in the denominator is new. The whole difficulty is choosing the right one, so the cases are set out together.

3.5.1 Two means, \(\sigma _1\) and \(\sigma _2\) known

\[Z=\frac {(\overline {X}_1-\overline {X}_2)-(\mu _1-\mu _2)_0} {\sqrt {\dfrac {\sigma _1^2}{n_1}+\dfrac {\sigma _2^2}{n_2}}}\] where \((\mu _1-\mu _2)_0\) is the difference asserted by \(H_0\), almost always zero. As with a single mean, \(Z\) is used because \(\sigma \) is known, whatever the sample sizes. For large samples with \(\sigma \) unknown, \(\hat {S}_1\) and \(\hat {S}_2\) may replace them.

3.5.2 Two means, \(\sigma _1=\sigma _2\) unknown — the pooled \(t\)-test

\[t=\frac {(\overline {X}_1-\overline {X}_2)-(\mu _1-\mu _2)_0} {S_p\sqrt {\dfrac {1}{n_1}+\dfrac {1}{n_2}}}, \hspace {0.5cm} S_p^2=\frac {(n_1-1)\hat {S}_1^2+(n_2-1)\hat {S}_2^2}{n_1+n_2-2},\] on \(n_1+n_2-2\) degrees of freedom. The two samples are pooled into one estimate of the common variance, each weighted by its own degrees of freedom, so the larger sample counts for more. Both populations must be approximately normal, and the equal-variance assumption should be checked by the \(F\)-test below.

3.5.3 Paired samples — the paired \(t\)-test

When the same units are measured twice, the two samples are not independent and the tests above do not apply. Work with the \(n\) differences \(d=X_1-X_2\): \[t=\frac {\overline {d}-\mu _{d0}}{\hat {S}_d/\sqrt {n}}\] on \(n-1\) degrees of freedom. This is the one-sample \(t\)-test applied to the differences, and nothing more.

Pairing is not a complication to be tolerated but the reason such experiments are run. If the units differ greatly among themselves — twelve cars, eight typists, fourteen students matched on ability — that variation is irrelevant to the comparison, and differencing removes it entirely. Treating paired data as two independent samples leaves it inside the standard error, where it can hide the very effect the experiment was designed to find.

Which test applies is decided by the design, never by the data: ask whether each observation in the first sample is tied to one particular observation in the second.

3.5.4 Two proportions

\[Z=\frac {\hat {P}_1-\hat {P}_2} {\sqrt {\hat {P}(1-\hat {P})\left (\dfrac {1}{n_1}+\dfrac {1}{n_2}\right )}}, \hspace {0.5cm} \hat {P}=\frac {X_1+X_2}{n_1+n_2},\] where \(X_1\) and \(X_2\) are the numbers of successes. The pooled estimate \(\hat {P}\) is used because \(H_0\) asserts that the two proportions are one and the same number, and every quantity in a test is computed under \(H_0\). This is exactly where the test differs from the confidence interval for \(P_1-P_2\), which assumes no equality and lets each sample supply its own \(\hat {P}\).

3.5.5 Two variances — the \(F\)-test

\[F=\frac {\hat {S}_1^2}{\hat {S}_2^2}\sim F_{v_1,\,v_2},\qquad v_1=n_1-1,\quad v_2=n_2-1,\] under \(H_0:\sigma _1^2=\sigma _2^2\). Both populations must be normal — the \(F\)-test is notably sensitive to that, more so than the \(t\)-test is.

Choosing the critical value needs care, since tables usually print only the upper tail:

\(\bullet \)

\(H_1:\sigma _1^2>\sigma _2^2\) — reject if \(F>F_{\alpha ,\,v_1,v_2}\).

\(\bullet \)

\(H_1:\sigma _1^2<\sigma _2^2\) — the critical region is the lower tail: reject if \(F<F_{1-\alpha ,\,v_1,v_2}=1/F_{\alpha ,\,v_2,v_1}\), the degrees of freedom swapping.

\(\bullet \)

\(H_1:\sigma _1^2\neq \sigma _2^2\) — reject if \(F>F_{\alpha /2,\,v_1,v_2}\) or \(F<1/F_{\alpha /2,\,v_2,v_1}\).

For a two-tailed test there is a standard shortcut: put the larger sample variance on top, so that \(F\geq 1\) automatically, and compare with the upper critical value alone.

This test has two uses. It answers questions about variability in their own right — is one machine more consistent than another — and it is the check that licenses the pooled \(t\)-test of the previous case. Where both are wanted, the \(F\)-test is done first.

3.5.6 Choosing between them

One warning that applies to all of them. A test that fails to reject \(H_0\) has not shown the two populations to be the same; it has shown that this experiment could not tell them apart. With small samples that is usually a statement about the sample size rather than about the populations, and the confidence interval — which displays how large the difference might still be — is the more honest way to report it.