8.6 Testing Many Series at Once

Trend tests are rarely run once. A study will test every rain gauge in a country or every grid cell in a region, and at the \(5\%\) level one test in twenty will be significant with no trend present. Testing \(200\) stations yields about ten false positives before any real signal is counted.

The Bonferroni correction of Chapter 3 — compare each \(p\)-value against \(\alpha /m\) — controls the probability of any false positive, but at \(m = 200\) it is so severe that genuine trends are missed. The modern alternative is the Benjamini–Hochberg procedure, which controls instead the expected proportion of false discoveries among those declared significant.

Order the \(p\)-values \(p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}\), find \[k = \max \left \{ i \ :\ p_{(i)} \leq \dfrac {i}{m}\,\alpha \right \},\] and reject the hypotheses attached to \(p_{(1)}, \ldots , p_{(k)}\). If no \(i\) satisfies the inequality, reject nothing. The false discovery rate is then at most \(\alpha \).

Bonferroni asks ”how confident am I that I have made no mistake at all?” Benjamini–Hochberg asks ”of the findings I am reporting, what fraction are likely to be wrong?” For a map of two hundred stations the second question is almost always the useful one.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.