6.1 Variance-stabilizing transformations
Non-constant variance usually arises because the variance of \(y\) is a function of its mean. Counts have variance equal to their mean; measurements subject to a percentage error have a standard deviation proportional to the mean; proportions have variance \(\pi (1 - \pi )/n\). In each case the remedy is a function \(g\) such that \(g(y)\) has (approximately) constant variance. The tool for finding it is the following approximation.
Lemma 6.1.1 (Delta method). Let \(y\) have mean \(\mu \) and small variance \(\sigma ^2(\mu )\), and let \(g\) be differentiable at \(\mu \). Then, approximately, \[ \operatorname {Var}\bigl (g(y)\bigr ) \approx g'(\mu )^2\,\sigma ^2(\mu ) . \]
Derivation. By Taylor’s theorem, \(g(y) = g(\mu ) + g'(\mu )(y - \mu ) + R\), where the remainder \(R\) is of order \((y - \mu )^2\). When \(y\) stays close to \(\mu \) the remainder is negligible, and \(g(y) \approx g(\mu ) + g'(\mu )(y - \mu )\). The right side is a linear function of \(y\), with variance \(g'(\mu )^2\operatorname {Var}(y)\). □
To make \(\operatorname {Var}(g(y))\) constant we need \(g'(\mu ) \propto 1/\sigma (\mu )\), that is, \[ g(\mu ) = \int \frac {d\mu }{\sigma (\mu )} . \]
Theorem 6.1.2. If the standard deviation of \(y\) is proportional to a power of its mean, \(\sigma (\mu ) \propto \mu ^a\), then \(g(y) = y^{1 - a}\) for \(a \neq 1\), and \(g(y) = \ln y\) for \(a = 1\), has approximately constant variance. If \(y\) is a binomial proportion, with \(\sigma ^2(\pi ) = \pi (1 - \pi )/n\), then \(g(y) = \arcsin \sqrt {y}\) has approximately constant variance \(1/(4n)\).
Proof. With \(\sigma (\mu ) = c\mu ^a\), \(\int \mu ^{-a}\,d\mu = \mu ^{1 - a}/(1 - a)\) for \(a \neq 1\) and \(\ln \mu \) for \(a = 1\); constant multiples do not matter. Check by Lemma 6.1.1: for \(g(y) = y^{1-a}\), \(g'(\mu )^2\sigma ^2(\mu ) = (1 - a)^2\mu ^{-2a}c^2\mu ^{2a} = (1 - a)^2c^2\), and for \(g = \ln \), \(\mu ^{-2}c^2\mu ^2 = c^2\), both free of \(\mu \).
For a proportion, let \(g(\pi ) = \arcsin \sqrt \pi \). By the chain rule \[ g'(\pi ) = \frac {1}{\sqrt {1 - \pi }}\cdot \frac {1}{2\sqrt \pi } = \frac {1}{2\sqrt {\pi (1 - \pi )}} , \] so \(g'(\pi )^2\sigma ^2(\pi ) = \dfrac {1}{4\pi (1 - \pi )}\cdot \dfrac {\pi (1 - \pi )}{n} = \dfrac {1}{4n}\). □
The common cases are:
| Relation of \(\sigma ^2\) to \(E(y)\) | Typical data | Transformation |
| \(\sigma ^2 \propto \) constant | none | |
| \(\sigma ^2 \propto E(y)\) | counts (Poisson) | \(\sqrt {y}\) |
| \(\sigma ^2 \propto E(y)(1 - E(y))\) | proportions | \(\arcsin \sqrt {y}\) |
| \(\sigma ^2 \propto [E(y)]^2\) | percentage errors | \(\ln y\) |
| \(\sigma ^2 \propto [E(y)]^3\) | \(y^{-1/2}\) | |
| \(\sigma ^2 \propto [E(y)]^4\) | \(y^{-1}\) |
The strength of the transformation increases down the table. When the response varies over less than a factor of about two or three, the transformations hardly change anything; when \(y_{\max }/y_{\min }\) is large, they matter a great deal. The form of \(\sigma ^2(\mu )\) may be known from the nature of the data, or it may be suggested by the funnel in a plot of residuals against fitted values (Figure 9).
Remark. A model fitted to \(g(y)\) predicts \(g(y)\). Converting back to the original scale needs care. Because \(g\) is monotone, an interval for \(g(y)\) converts to an interval for \(y\) by applying \(g^{-1}\) to its end points, and a prediction interval keeps its probability. But \(g^{-1}\bigl (\widehat {g(y)}\bigr )\) is not an estimate of the mean of \(y\): for \(g = \ln \), \(E(\ln y) \neq \ln E(y)\). If \(\ln y\) is normal with mean \(m\) and variance \(s^2\), then \(e^m\) is the median of \(y\), and its mean is \(e^{m + s^2/2}\), somewhat larger.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.