1.7 Ancillary Statistics

Information about \(\theta \) comes from the way the distribution of a statistic changes with \(\theta \). A statistic whose distribution does not change with \(\theta \) at all would seem to carry no information about it.

Definition 1.7.1. A statistic \(U(X)\) is ancillary if its distribution does not depend on \(\theta \).

For example, if \(X_1, \dots , X_n\) is a sample from a location family \(f(x - \theta )\), then \(X_i - \theta \) has a distribution free of \(\theta \), and so does any function of the differences \(X_i - X_j\), such as the range \(X_{(n)} - X_{(1)}\) or \(S^2\). For a scale family \(\frac 1\theta f\left (\frac x\theta \right )\), ratios such as \(\frac {X_1}{X_2}\) are ancillary.

The next theorem uses completeness and ancillarity to prove that two statistics are independent without finding their joint distribution.

Theorem 1.7.2 (Basu). If \(T(X)\) is a complete sufficient statistic, then \(T(X)\) is independent of every ancillary statistic \(U(X)\).

Proof. We must show that \(P_\theta (U\in B, T\in C) = P_\theta (U\in B)P_\theta (T\in C)\) for all sets \(B\), \(C\) and all \(\theta \). Fix \(B\) and define \[g(t) = P(U\in B\mid T = t) - P(U\in B) .\] The first term does not depend on \(\theta \) by sufficiency of \(T\), and the second does not by ancillarity of \(U\), so \(g(T)\) is a statistic. By the law of total expectation, \[E_\theta \left [g(T)\right ] = E_\theta \left [P(U\in B\mid T)\right ] - P(U\in B) = P(U\in B) - P(U\in B) = 0\] for every \(\theta \). Completeness gives \(g(T) = 0\) with probability one, that is, \(P(U\in B\mid T = t) = P(U\in B)\) for almost all \(t\). Hence, writing \(f_T(t;\theta )\) for the density of \(T\), \begin {align*} P_\theta (U\in B, T\in C) &= \int _C P(U\in B\mid T = t)f_T(t;\theta )\,dt\\ &= P(U\in B)\int _C f_T(t;\theta )\,dt = P(U\in B)P_\theta (T\in C), \end {align*}

with a sum in place of the integral in the discrete case. □

Example 1.7.3. Let \(X_1, \dots , X_n\) be a random sample from the exponential distribution with mean \(\theta \). Show that \(T = \sum X_i\) and \(U = \left (\frac {X_1}{T}, \dots , \frac {X_n}{T}\right )\) are independent, and use this to find \(E(X_1\mid T)\).

Solution. The variables \(Z_i = \frac {X_i}{\theta }\) are independent \(\text {EXP}(1)\), whose distribution is free of \(\theta \). Since \[\frac {X_i}{T} = \frac {X_i/\theta }{\sum _j X_j/\theta } = \frac {Z_i}{\sum _j Z_j},\] \(U\) is a function of \((Z_1, \dots , Z_n)\) and is ancillary. By example 1.6.16, \(T\) is complete and sufficient. By theorem 1.7.2, \(T\) and \(U\) are independent.

Now \(X_1 = \frac {X_1}{T}\cdot T\). Taking expectations and using independence, \(\theta = E(X_1) = E\left (\frac {X_1}{T}\right )E(T) = E\left (\frac {X_1}{T}\right )n\theta \), so \(E\left (\frac {X_1}{T}\right ) = \frac 1n\). Conditioning on \(T\), and again using independence, \[E(X_1\mid T) = T\,E\left (\frac {X_1}{T} \mid T\right ) = T\,E\left (\frac {X_1}{T}\right ) = \frac Tn = \bar X .\]

Example 1.7.4. Let \(X_1, \dots , X_n\) be a random sample from \(N(\mu , \sigma ^2)\). Prove that \(\bar X\) and \(S^2\) are independent.

Solution. Fix \(\sigma ^2\) and regard \(\mu \) as the only parameter. The family \(N(\mu , \sigma ^2)\) with \(\sigma ^2\) known is a REF with \(\eta = \frac {\mu }{\sigma ^2}\), so \(\bar X\) is complete and sufficient for \(\mu \). Write \(Z_i = X_i - \mu \), which are \(N(0, \sigma ^2)\) with distribution free of \(\mu \). Then \(X_i - \bar X = Z_i - \bar Z\), so \[S^2 = \frac {1}{n - 1}\sum (X_i - \bar X)^2 = \frac {1}{n - 1}\sum (Z_i - \bar Z)^2\] is a function of the \(Z_i\) and is ancillary for \(\mu \). By theorem 1.7.2, \(\bar X\) and \(S^2\) are independent when the true variance is \(\sigma ^2\). Since \(\sigma ^2\) was arbitrary, they are independent for all \(\mu \) and \(\sigma ^2\).

Problem 1.7.1. Let \(X_1, \, \cdots \, , \, X_n\) be a random sample from the \(GAM(\alpha , \beta )\) distribution. Show that \(T(X) = \sum ^n_{i = 1} X_i\) and \(U(X) = \left (\frac {X_1}{T}, \, \cdots \, , \, \frac {X_n}{T}\right )\) are independent random variables. Find \(E\left (\frac {X_1}{T}\right )\).

Show solution

Solution.

Independence

Treat \(\alpha \) as known and \(\beta \) as the parameter. The model is then a one-parameter exponential family with natural statistic \(T=\sum _i X_i\), so \(T\) is complete and sufficient for \(\beta \).

The vector \(U\) is ancillary: replacing each \(X_i\) by \(cX_i\) for a constant \(c>0\) leaves every ratio \(X_i/T\) unchanged, and such a rescaling is exactly what changes one value of \(\beta \) into another. So the distribution of \(U\) does not depend on \(\beta \) at all. By Basu’s theorem a complete sufficient statistic is independent of every ancillary statistic, hence \(T\) and \(U\) are independent.

The expectation

The components of \(U\) sum to one: \[\sum _{i=1}^{n}\frac {X_i}{T} = \frac {T}{T} = 1 .\] Taking expectations, \(\sum _{i=1}^{n}E\left (X_i/T\right ) = 1\). The \(X_i\) are identically distributed and enter \(T\) symmetrically, so all \(n\) terms are equal, giving \[E\left (\frac {X_1}{T}\right ) = \frac {1}{n}.\]

Note. The expectation was obtained without integrating anything. Symmetry plus the fact that the ratios sum to one does all the work — a pattern worth remembering, because the direct calculation requires the Dirichlet distribution of \(U\) and is far longer.

Further reading

1.
Fisher, R. A. (1922), On the mathematical foundations of theoretical statistics, Philosophical Transactions of the Royal Society A, 222, 309–368.
2.
Koopman, B. O. (1936), On distributions admitting a sufficient statistic, Transactions of the American Mathematical Society, 39, 399–409.
3.
Rao, C. R. (1945), Information and the accuracy attainable in the estimation of statistical parameters, Bulletin of the Calcutta Mathematical Society, 37, 81–91.
4.
Blackwell, D. (1947), Conditional expectation and unbiased sequential estimation, Annals of Mathematical Statistics, 18, 105–110.
5.
Lehmann, E. L., and Scheffé, H. (1950), Completeness, similar regions, and unbiased estimation, part I, Sankhyā, 10, 305–340.
6.
Basu, D. (1955), On statistics independent of a complete sufficient statistic, Sankhyā, 15, 377–380.
7.
Bahadur, R. R. (1957), On unbiased estimates of uniformly minimum variance, Sankhyā, 18, 211–224.
8.
Boos, D. D., and Hughes-Oliver, J. M. (1998), Applications of Basu’s theorem, The American Statistician, 52, 218–221.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.