3.6 Variance-stabilising transformations
When the error variance changes with the mean, a transformation of the response can often make it constant. The guide is the delta method.
Proposition 3.6.1. Suppose \(E(Y) = \mu \) and the standard deviation of \(Y\) is proportional to a power of the mean, \(\sigma _Y = \theta \mu ^\alpha \). For a smooth function \(g\), \[ \Var \bigl (g(Y)\bigr ) \approx g'(\mu )^2\,\theta ^2\mu ^{2\alpha } , \] which is constant in \(\mu \) when \(g(y) = y^{1-\alpha }\) for \(\alpha \ne 1\) and \(g(y) = \log y\) for \(\alpha = 1\).
Proof. By Taylor’s theorem, \(g(Y) \approx g(\mu ) + g'(\mu )(Y - \mu )\), so \(\Var \bigl (g(Y)\bigr ) \approx g'(\mu )^2\Var (Y) = g'(\mu )^2\theta ^2\mu ^{2\alpha }\). This does not depend on \(\mu \) when \(g'(\mu ) \propto \mu ^{-\alpha }\). Integrating, \(g(\mu ) \propto \mu ^{1-\alpha }\) if \(\alpha \ne 1\) and \(g(\mu ) = \log \mu \) if \(\alpha = 1\). □
| relationship | \(\alpha \) | transformation |
| \(\sigma _Y\) constant | \(0\) | none |
| \(\sigma _Y^2 \propto \mu \) (Poisson counts) | \(\frac 12\) | square root, \(\sqrt y\) |
| \(\sigma _Y \propto \mu \) (constant coefficient of variation) | \(1\) | logarithm, \(\log y\) |
| \(\sigma _Y \propto \mu ^{3/2}\) | \(\frac 32\) | reciprocal square root |
| \(\sigma _Y \propto \mu ^2\) | \(2\) | reciprocal, \(1/y\) |
For a binomial proportion \(\hat p\) based on \(m\) trials, \(\Var (\hat p) = p(1 - p)/m\), which is not a power of the mean, and the same argument gives the arcsine transformation \(\arcsin \sqrt {\hat p}\), with variance close to \(1/(4m)\) whatever \(p\).
The exponent \(\alpha \) can be estimated from the data. Taking logarithms of \(\sigma _i = \theta \mu _i^\alpha \) gives \(\log \sigma _i = \log \theta + \alpha \log \mu _i\), so the slope of a straight line fitted to the points \((\log \bar y_{i\cdot },\ \log s_i)\) estimates \(\alpha \).
Example 3.6.2. Four insecticides were each applied to five randomly chosen plots, and the number of surviving insects counted on each plot:
| insecticide | \(\bar y_{i\cdot }\) | \(s_i\) | |||||
| \(1\) | \(4\) | \(7\) | \(3\) | \(6\) | \(5\) | \(5\) | \(1.58\) |
| \(2\) | \(12\) | \(18\) | \(15\) | \(11\) | \(19\) | \(15\) | \(3.54\) |
| \(3\) | \(34\) | \(47\) | \(39\) | \(31\) | \(49\) | \(40\) | \(7.87\) |
| \(4\) | \(78\) | \(101\) | \(92\) | \(70\) | \(109\) | \(90\) | \(16.05\) |
The standard deviation grows with the mean, and Bartlett’s test on the raw counts gives \(\chi _0^2 = 16.61\) on \(3\) degrees of freedom (\(p = 0.0008\)). The fitted slope of \(\log s_i\) on \(\log \bar y_{i\cdot }\) is \(\hat \alpha = 0.80\), which suggests \(y^{0.2}\), between the square root (\(\alpha = \frac 12\), the Poisson value) and the logarithm (\(\alpha = 1\)). Both remove the problem:
| scale | Bartlett \(p\) | ||||
| \(y\) | \(1.58\) | \(3.54\) | \(7.87\) | \(16.05\) | \(0.0008\) |
| \(\sqrt y\) | \(0.36\) | \(0.46\) | \(0.62\) | \(0.85\) | \(0.40\) |
| \(\log y\) | \(0.33\) | \(0.24\) | \(0.20\) | \(0.18\) | \(0.64\) |
The logarithm stabilises the spread best, which agrees with \(\hat \alpha \) being nearer \(1\) than \(\frac 12\): these counts are more variable than Poisson counts would be, as insect counts usually are. On the log scale the analysis of variance gives \(F_0 = 131.1\) on \(3\) and \(16\) degrees of freedom, and conclusions are stated about the means of \(\log y\), that is, about ratios of typical counts.
Remark. The choice of \(\lambda \) in \(y^\lambda \) can be made formally by the method of Box and Cox, which maximises the likelihood of the transformed model over \(\lambda \). In practice the analyst rounds to a value with an interpretation, \(1\), \(\frac 12\), \(0\) (the logarithm) or \(-1\), and the plot of \(\log s_i\) against \(\log \bar y_{i\cdot }\) usually suffices. A transformation chosen for constant variance often improves normality as well.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.