4.1 Comparing Two Tests
Any sensible comparison must hold something fixed. Two tests can be compared at the same significance level and the same alternative by asking how many observations each needs to attain the same power.
Definition 4.1.1 (Relative efficiency). Let \(T_1\) and \(T_2\) be two tests of the same \(H_0\) against the same alternative, both at level \(\alpha \). If \(n_1\) and \(n_2\) observations respectively are required to attain power \(\beta \), the relative efficiency of \(T_1\) with respect to \(T_2\) is \[\mathrm {RE}\left (T_1,T_2\right ) = \frac {n_2}{n_1}.\]
A value greater than one means \(T_1\) needs fewer observations and is the better test; a value of \(\tfrac 12\) means \(T_1\) needs twice the data to do the same work.
This quantity is awkward as it stands: it depends on \(\alpha \), on \(\beta \), on the particular alternative, and on the sample size. The standard resolution is to let the alternative approach the null as the sample size grows, at the rate that keeps the power fixed and strictly between \(\alpha \) and \(1\).
Definition 4.1.2 (Pitman asymptotic relative efficiency). Consider a sequence of local alternatives \(\theta _n = \theta _0 + \delta /\sqrt {n}\) for fixed \(\delta \neq 0\). The asymptotic relative efficiency of \(T_1\) with respect to \(T_2\) is \[\mathrm {ARE}\left (T_1,T_2\right ) = \lim _{n\to \infty }\frac {n_2}{n_1},\] when the limit exists.
Note 4.1.3. The \(1/\sqrt {n}\) rate is not arbitrary. Under a fixed alternative the power of any reasonable test tends to \(1\), and all tests eventually look equally good, so no comparison is possible. Alternatives receding faster than \(1/\sqrt {n}\) send the power down to \(\alpha \) and again permit no comparison. The rate \(1/\sqrt {n}\) is the one at which power settles at a constant strictly between \(\alpha \) and \(1\), and it is therefore the only scale on which a non-trivial comparison survives in the limit.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.