2.3 Unequal variances
If the two treatments may have different variances, \(\varepsilon _{1j} \sim N(0, \sigma _1^2)\) and \(\varepsilon _{2j} \sim N(0, \sigma _2^2)\), there is no exact \(t\) test. The natural statistic is \[ t_0^* = \frac {\bar y_1 - \bar y_2}{\sqrt {s_1^2/n_1 + s_2^2/n_2}} , \] whose numerator has variance \(\sigma _1^2/n_1 + \sigma _2^2/n_2\), estimated without pooling. Its null distribution is approximately \(t_\nu \), where \(\nu \) is found as follows.
Proposition 2.3.1 (Satterthwaite’s approximation). Let \(V = a_1S_1^2 + a_2S_2^2\) with \(a_i > 0\), where \((n_i - 1)S_i^2/\sigma _i^2 \sim \chi ^2_{n_i-1}\) independently. If \(V\) is approximated by a multiple of a chi-square variable, \(\nu V/E(V) \approx \chi ^2_\nu \), with mean and variance matched, then \[ \nu = \frac {\bigl (a_1\sigma _1^2 + a_2\sigma _2^2\bigr )^2}{\dfrac {(a_1\sigma _1^2)^2}{n_1 - 1} + \dfrac {(a_2\sigma _2^2)^2}{n_2 - 1}} . \]
Proof. \(E(S_i^2) = \sigma _i^2\) and \(\Var (S_i^2) = \sigma _i^4\cdot 2(n_i - 1)/(n_i - 1)^2 = 2\sigma _i^4/(n_i - 1)\), so \[ E(V) = a_1\sigma _1^2 + a_2\sigma _2^2 , \qquad \Var (V) = \frac {2(a_1\sigma _1^2)^2}{n_1 - 1} + \frac {2(a_2\sigma _2^2)^2}{n_2 - 1} . \] If \(\nu V/E(V) \sim \chi ^2_\nu \) exactly, its mean is \(\nu \) (automatically) and its variance \(\nu ^2\Var (V)/E(V)^2\) must equal \(2\nu \), that is, \(\nu = 2E(V)^2/\Var (V)\), which is the formula stated. □
With \(a_i = 1/n_i\) and the unknown \(\sigma _i^2\) replaced by \(s_i^2\), this gives the Welch degrees of freedom \[ \nu = \frac {\bigl (s_1^2/n_1 + s_2^2/n_2\bigr )^2}{\dfrac {(s_1^2/n_1)^2}{n_1 - 1} + \dfrac {(s_2^2/n_2)^2}{n_2 - 1}} . \] It always lies between \(\min (n_1, n_2) - 1\) and \(n_1 + n_2 - 2\).
Example 2.3.2. For the catalyst data, \(s_1^2/n_1 = 0.7093\) and \(s_2^2/n_2 = 0.3411\). Because \(n_1 = n_2\), the Welch statistic is the same as the pooled one, \(t_0^* = 2.159\), but the degrees of freedom are \[ \nu = \frac {(0.7093 + 0.3411)^2}{0.7093^2/7 + 0.3411^2/7} = 12.47 , \] and the \(p\)-value is \(0.051\). The two analyses straddle the \(5\%\) line. This should not be read as a contradiction: both say that the evidence is borderline. A decision that turns on the third decimal of a \(p\)-value is not a decision the data support, and the confidence interval is the better summary.
Note. With equal sample sizes the pooled test is robust to unequal variances: its level is close to nominal even when \(\sigma _1^2 \ne \sigma _2^2\). With unequal sample sizes it is not. If the smaller sample comes from the more variable population, the pooled variance is too small and the test rejects too often. This is one reason, beyond simplicity, to design experiments with equal replication.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.