1.7 Ancillary Statistics
Information about \(\theta \) comes from the way the distribution of a statistic changes with \(\theta \). A statistic whose distribution does not change with \(\theta \) at all would seem to carry no information about it.
Definition 1.7.1. A statistic \(U(X)\) is ancillary if its distribution does not depend on \(\theta \).
For example, if \(X_1, \dots , X_n\) is a sample from a location family \(f(x - \theta )\), then \(X_i - \theta \) has a distribution free of \(\theta \), and so does any function of the differences \(X_i - X_j\), such as the range \(X_{(n)} - X_{(1)}\) or \(S^2\). For a scale family \(\frac 1\theta f\left (\frac x\theta \right )\), ratios such as \(\frac {X_1}{X_2}\) are ancillary.
The next theorem uses completeness and ancillarity to prove that two statistics are independent without finding their joint distribution.
Theorem 1.7.2 (Basu). If \(T(X)\) is a complete sufficient statistic, then \(T(X)\) is independent of every ancillary statistic \(U(X)\).
Proof. We must show that \(P_\theta (U\in B, T\in C) = P_\theta (U\in B)P_\theta (T\in C)\) for all sets \(B\), \(C\) and all \(\theta \). Fix \(B\) and define \[g(t) = P(U\in B\mid T = t) - P(U\in B) .\] The first term does not depend on \(\theta \) by sufficiency of \(T\), and the second does not by ancillarity of \(U\), so \(g(T)\) is a statistic. By the law of total expectation, \[E_\theta \left [g(T)\right ] = E_\theta \left [P(U\in B\mid T)\right ] - P(U\in B) = P(U\in B) - P(U\in B) = 0\] for every \(\theta \). Completeness gives \(g(T) = 0\) with probability one, that is, \(P(U\in B\mid T = t) = P(U\in B)\) for almost all \(t\). Hence, writing \(f_T(t;\theta )\) for the density of \(T\), \begin {align*} P_\theta (U\in B, T\in C) &= \int _C P(U\in B\mid T = t)f_T(t;\theta )\,dt\\ &= P(U\in B)\int _C f_T(t;\theta )\,dt = P(U\in B)P_\theta (T\in C), \end {align*}
with a sum in place of the integral in the discrete case. □
Example 1.7.3. Let \(X_1, \dots , X_n\) be a random sample from the exponential distribution with mean \(\theta \). Show that \(T = \sum X_i\) and \(U = \left (\frac {X_1}{T}, \dots , \frac {X_n}{T}\right )\) are independent, and use this to find \(E(X_1\mid T)\).
Solution. The variables \(Z_i = \frac {X_i}{\theta }\) are independent \(\text {EXP}(1)\), whose distribution is free of \(\theta \). Since \[\frac {X_i}{T} = \frac {X_i/\theta }{\sum _j X_j/\theta } = \frac {Z_i}{\sum _j Z_j},\] \(U\) is a function of \((Z_1, \dots , Z_n)\) and is ancillary. By example 1.6.16, \(T\) is complete and sufficient. By theorem 1.7.2, \(T\) and \(U\) are independent.
Now \(X_1 = \frac {X_1}{T}\cdot T\). Taking expectations and using independence, \(\theta = E(X_1) = E\left (\frac {X_1}{T}\right )E(T) = E\left (\frac {X_1}{T}\right )n\theta \), so \(E\left (\frac {X_1}{T}\right ) = \frac 1n\). Conditioning on \(T\), and again using independence, \[E(X_1\mid T) = T\,E\left (\frac {X_1}{T} \mid T\right ) = T\,E\left (\frac {X_1}{T}\right ) = \frac Tn = \bar X .\]
Example 1.7.4. Let \(X_1, \dots , X_n\) be a random sample from \(N(\mu , \sigma ^2)\). Prove that \(\bar X\) and \(S^2\) are independent.
Solution. Fix \(\sigma ^2\) and regard \(\mu \) as the only parameter. The family \(N(\mu , \sigma ^2)\) with \(\sigma ^2\) known is a REF with \(\eta = \frac {\mu }{\sigma ^2}\), so \(\bar X\) is complete and sufficient for \(\mu \). Write \(Z_i = X_i - \mu \), which are \(N(0, \sigma ^2)\) with distribution free of \(\mu \). Then \(X_i - \bar X = Z_i - \bar Z\), so \[S^2 = \frac {1}{n - 1}\sum (X_i - \bar X)^2 = \frac {1}{n - 1}\sum (Z_i - \bar Z)^2\] is a function of the \(Z_i\) and is ancillary for \(\mu \). By theorem 1.7.2, \(\bar X\) and \(S^2\) are independent when the true variance is \(\sigma ^2\). Since \(\sigma ^2\) was arbitrary, they are independent for all \(\mu \) and \(\sigma ^2\).
Problem 1.7.1. Let \(X_1, \, \cdots \, , \, X_n\) be a random sample from the \(GAM(\alpha , \beta )\) distribution. Show that \(T(X) = \sum ^n_{i = 1} X_i\) and \(U(X) = \left (\frac {X_1}{T}, \, \cdots \, , \, \frac {X_n}{T}\right )\) are independent random variables. Find \(E\left (\frac {X_1}{T}\right )\).
Show solution
Solution.
Independence
Treat \(\alpha \) as known and \(\beta \) as the parameter. The model is then a one-parameter exponential family with natural statistic \(T=\sum _i X_i\), so \(T\) is complete and sufficient for \(\beta \).
The vector \(U\) is ancillary: replacing each \(X_i\) by \(cX_i\) for a constant \(c>0\) leaves every ratio \(X_i/T\) unchanged, and such a rescaling is exactly what changes one value of \(\beta \) into another. So the distribution of \(U\) does not depend on \(\beta \) at all. By Basu’s theorem a complete sufficient statistic is independent of every ancillary statistic, hence \(T\) and \(U\) are independent.
The expectation
The components of \(U\) sum to one: \[\sum _{i=1}^{n}\frac {X_i}{T} = \frac {T}{T} = 1 .\] Taking expectations, \(\sum _{i=1}^{n}E\left (X_i/T\right ) = 1\). The \(X_i\) are identically distributed and enter \(T\) symmetrically, so all \(n\) terms are equal, giving \[E\left (\frac {X_1}{T}\right ) = \frac {1}{n}.\]
Note. The expectation was obtained without integrating anything. Symmetry plus the fact that the ratios sum to one does all the work — a pattern worth remembering, because the direct calculation requires the Dirichlet distribution of \(U\) and is far longer.
Further reading
- 1.
- Fisher, R. A. (1922), On the mathematical foundations of theoretical statistics, Philosophical Transactions of the Royal Society A, 222, 309–368.
- 2.
- Koopman, B. O. (1936), On distributions admitting a sufficient statistic, Transactions of the American Mathematical Society, 39, 399–409.
- 3.
- Rao, C. R. (1945), Information and the accuracy attainable in the estimation of statistical parameters, Bulletin of the Calcutta Mathematical Society, 37, 81–91.
- 4.
- Blackwell, D. (1947), Conditional expectation and unbiased sequential estimation, Annals of Mathematical Statistics, 18, 105–110.
- 5.
- Lehmann, E. L., and Scheffé, H. (1950), Completeness, similar regions, and unbiased estimation, part I, Sankhyā, 10, 305–340.
- 6.
- Basu, D. (1955), On statistics independent of a complete sufficient statistic, Sankhyā, 15, 377–380.
- 7.
- Bahadur, R. R. (1957), On unbiased estimates of uniformly minimum variance, Sankhyā, 18, 211–224.
- 8.
- Boos, D. D., and Hughes-Oliver, J. M. (1998), Applications of Basu’s theorem, The American Statistician, 52, 218–221.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.