3.3 Type I and Type II Errors
When we test a null hypothesis by applying a significant test to a sample, we can never be certain that
our conclusion is correct.
All we know is that our decision is probably correct.
That is when we reject \(H_0\), it is because our statistic lies in the CR and this event is unlikely if the null
hypothesis is true we can make two types of errors.
Type I error: The null hypothesis is rejected when in fact it is true \((\alpha )\).
Type II error: The null hypothesis is retained when in fact it is false \((\beta )\).
The probability of type II error can not be generally calculated because it depends on the population
mean which is unknown. \(\beta \) can be computed if \(\mu \), \(\sigma ^2\), \(n\) are given.
The power of a hypothesis test is simply \(1-\beta \). And this is basically (power of a test) is the probability that
we make the right decision when the null hypothesis is not correct (we correctly reject
\(H_0\)).
Example 3.9. A sample of \(n=100\) is drawn from a population with \(\sigma ^2=10\,000\), to test
\[H_0: \mu \geq 30\hspace {1cm}\text {against}\hspace {1cm}H_1:\mu <30\]
at \(\alpha =5\%\). Find the probability of a Type II error, and the power of the test, when the true mean is
in fact \(\mu =26\).
Solution. The population standard deviation is \(\sigma =\sqrt {10\,000}=100\), so the standard error of the mean is \[\sigma _{\overline {X}}=\frac {\sigma }{\sqrt {n}}=\frac {100}{\sqrt {100}}=10.\]
A Type II error is failing to reject \(H_0\) when it is false, so \(\beta \) can only be computed against a stated alternative value of \(\mu \) — here \(\mu =26\). The calculation runs in two stages: first find the sample mean at which the decision changes, then find how often the true distribution lands on the wrong side of it.
Stage 1: the decision boundary. The test is one-tailed to the left, so \(H_0\) is rejected for small \(\overline {X}\), and the critical value is \(-Z_{0.05}=-1.64\). Converting that back to a sample mean, under \(H_0\) with \(\mu =30\): \[Z_c=\frac {\overline {X}_c-\mu }{\sigma _{\overline {X}}} \implies -1.64=\frac {\overline {X}_c-30}{10} \implies \overline {X}_c=30-16.4=13.6.\] So \(H_0\) is rejected when \(\overline {X}<13.6\), and retained when \(\overline {X}>13.6\).
Stage 2: the probability of landing in the wrong region. If the truth is \(\mu =26\), \(H_0\) is false, and a Type II error occurs whenever the sample mean nonetheless exceeds \(13.6\): \begin {align*} \beta &=P\big (\overline {X}>13.6 \mid \mu =26,\ \sigma _{\overline {X}}=10\big )\\ &=P\!\left (Z>\frac {13.6-26}{10}\right )\\ &=P(Z>-1.24)\\ &=0.8925. \end {align*}
The power of the test is therefore \[1-\beta =1-0.8925=0.1075.\]
That power is dismal, and it is worth seeing why. With \(\sigma =100\), the shift from \(\mu =30\) to \(\mu =26\) is only \(0.4\) of a standard error. A test at \(\alpha =5\%\) has almost no chance of catching a difference that small: close to nine times in ten it will fail to reject a null hypothesis that is false. Increasing \(n\) is the usual remedy, since it shrinks \(\sigma _{\overline {X}}\) and moves the two distributions apart in standard-error terms.
Note also that \(\alpha \) and \(\beta \) pull against each other. Lowering \(\alpha \) to \(1\%\) would move \(\overline {X}_c\) further left, making a Type I error rarer and a Type II error still more likely. Only a larger sample improves both at once.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.