1.7 Ancillary Statistics

Information on the unknown parameter \(\theta \) is provided by the sensitivity of the distribution of a statistic to changes in the parameter.
A statistic that has no sensitivity at all in distribution to the parameter, which would appear to contain little information about \(\theta \), is called an ancillary statistics.

Definition 1.7.1. \(U(X)\) is an ancillary statistic if its distribution does not depend on the unknown parameter \(\theta \).

The following theorem allows us to use the properties of completeness and ancillarity to prove independence of two statistics without finding their joint distribution.

Theorem 1.7.2 (Basu). Let \(X\) have probability (density) function \(\{f_{\theta }(x);\, \theta \in \Omega \}\). Let \(T(X)\) be a complete sufficient statistic. Then \(T(X)\) is independent of every ancillary statistic \(U(X)\).

Proof. We need to show that \(P\left (U(X) \in B, \, T(X)\in C\right ) = P\left (U(x)\in B\right )\, P\left (T(X)\in C\right )\) for all sets \(B, C\) and \(\theta \in \Omega \).
Let \(g(t) = P\left (U(X) \in B / T(X) = t\right ) - P\left (U(X) \in B\right )\) for all \(t\in A\) where \(P\left (T\in A\right ) = 1\). By sufficiency \(P\left (U(X) \in B / T(X) = t\right )\) does not depend on \(\theta \) and by ancillarity \(P\left (U(X)\in B\right )\) also does not depend on \(\theta \). Therefore \(g(T)\) is a statistic.
Let \[I\left (U(X) \in B\right ) = \begin {cases} 1 & \text {if}\,\,\,\, U(X) \in B\\ 0 & \text {otherwise}\\ \end {cases}\] then \(E\left [I(U(X) \in B)\right ] = P\left (U(X)\in B\right )\), \(\, E\left [I(U(X) = B) / T = t \right ] = P\left (U(X) \in B / T = t\right )\) and \(g(t) = E\left [I\left (U(X) \in B / T = t\right )\right ] - E\left [I\left (U(X) \in B\right )\right ]\). Then \begin {align*} E\left [g(t)\right ] & = E\left \{E\left (I(U(X) \in B / T = t\right )\right \} - E\left [I\left (U(X) \in B\right )\right ]\\ & = E\left [E\left (U(X) \in B\right )\right ] - E\left [I\left (U(X) \in B\right )\right ]\\ & = 0. \end {align*}

Since \(T\) is complete this implies that \(P_{\theta }\left [g(T) = 0\right ] = 1\) for all \(\theta \in \Omega \). Therefore
\(P\left (U(X)\in B / T(X) = t\right ) = P\left (U(X) \in B\right )\) for all \(t \in A\) and \(B\).
Suppose \(T\) has probability density function \(g_{\theta }(t)\), then \begin {align*} P\left (U(X)\in B, T(X) \in C\right ) & = P\left (U(X)\in B / T(X) = t\right )\, P\left (T(X) \in C\right )\\ & = \int _C P\left (U(X) \in B / T = t\right )\, g_{\theta } (t) \, dt\\ & = \int _CP\left (U(X) \in B\right )\, g_{\theta }(t)\, dt\\ & = P\left (U(X)\in B\right )\, P\left (T(X)\in C\right ) \end {align*}

for all sets \(B, C\) and \(\theta \in \Omega \) as required. □

Example 1.7.3. Let \(X_1, \, \cdots \, , \, X_n\) be a random sample from the \(EXP(\theta )\) distribution. Show that \(T(X) = \sum ^n_{i = 1} X_i\) and \(U(X) = \left (\frac {X_1}{T}, \, \cdots \, , \, \frac {X_n}{T}\right )\) are independent random variables. Find \(E(X_1 \mid T)\).

Solution. \(X_i \, \thicksim \, EXP(\theta )\) \[T = \sum ^n_{i = 1} X_i \, \thicksim \, GAM(n,\theta )\] \[Z_i = \frac {X_i}{\theta } \, \thicksim \, EXP(1)\,\,\hspace {0.2cm}\text {is ancillary}.\] \[Z = \frac {T}{\theta } \, \thicksim \, GAM(n ,1)\, \, \text {is ancillary now}\] \[\frac {X_i}{T} = \frac {X_i/\theta }{T/\theta } = \frac {Z_i}{Z}\hspace {0.3cm}\text {is ancillary}\] \[\therefore \, \hspace {0.2cm} U(X) = \left (\frac {X_1}{T}, \, \cdots \, , \, \frac {X_n}{T}\right ) = \left (\frac {Z_1}{Z}, \, \cdots \, , \, \frac {Z_n}{Z}\right )\hspace {0.3cm} \text {is ancillary statistic}\] \(X_i\thicksim EXP(\theta )\) is a REF distribution.
\(\therefore \, T = \sum ^n_{i = 1} X_i\,\) is a complete sufficient statistic. Therefore by Basus’ theorem \(U(X)\) and \(T\) are independent.
Now \(\, E(X_1) = \theta \) \[E\left (\frac {X_1}{T}\, \cdot \, T\right ) = \theta \] \[E\left (\frac {X_1}{T}\right )\, E(T) = \theta \] \[E\left (\frac {X_1}{T}\right )\, n\theta = \theta \] \[\implies \hspace {0.3cm} E\left (\frac {X_1}{T}\right ) = \frac {1}{n}.\] □

Example 1.7.4. Let \(X_1, \, \cdots \, , \, X_n\) be a random sample from the \(N(\mu , \sigma ^2)\) distribution. Prove that \(\overline {X}\) and \(S^2\) are independent.

Solution. \(X_i \thicksim N(\mu , \sigma ^2)\) assume \(\sigma ^2\) is known, then \(\overline {X}\) is a complete sufficient statistic. \[Z_i = X_i - \mu \, \thicksim \, N(0, \sigma ^2)\,\hspace {0.2cm}\text {is ancillary}\] \(\overline {Z} = \overline {X} - \mu \, \thicksim \, N(0, \sigma ^2/n)\) is ancillary then \(\sum ^n_{i = 1} (Z_i - \overline {Z})^2\) is ancillary \[\sum ^n_{i = 1} (Z_i - \overline {Z})^2 = \sum ^n_{i = 1} (X_i - \mu - (\overline {X} - \mu )^2) = \sum ^n_{i = 1} (X_i - \overline {X})^2\,\hspace {0.2cm} \text {is ancillary}.\] Therefore \(\, S^2 = \frac {1}{n - 1}\sum ^n_{i = 1}(X_i - \overline {X})^2\) is ancillary.
Therefore by Basus’ theorem \(\overline {X}\) and \(S^2\) are independent. Since this argument holds for all values of \(\sigma ^2\).
\(\therefore \, \overline {X}\) and \(S^2\) are independent for all \(\mu \) and \(\sigma ^2\). □

Problem 1.7.1. Let \(X_1, \, \cdots \, , \, X_n\) be a random sample from the \(GAM(\alpha , \beta )\) distribution. Show that \(T(X) = \sum ^n_{i = 1} X_i\) and \(U(X) = \left (\frac {X_1}{T}, \, \cdots \, , \, \frac {X_n}{T}\right )\) are independent random variables. Find \(E\left (\frac {X_1}{T}\right )\).

Show solution

Solution.

Independence

Treat \(\alpha \) as known and \(\beta \) as the parameter. The model is then a one-parameter exponential family with natural statistic \(T=\sum _i X_i\), so \(T\) is complete and sufficient for \(\beta \).

The vector \(U\) is ancillary: replacing each \(X_i\) by \(cX_i\) for a constant \(c>0\) leaves every ratio \(X_i/T\) unchanged, and such a rescaling is exactly what changes one value of \(\beta \) into another. So the distribution of \(U\) does not depend on \(\beta \) at all. By Basu’s theorem a complete sufficient statistic is independent of every ancillary statistic, hence \(T\) and \(U\) are independent.

The expectation

The components of \(U\) sum to one: \[\sum _{i=1}^{n}\frac {X_i}{T} = \frac {T}{T} = 1 .\] Taking expectations, \(\sum _{i=1}^{n}E\left (X_i/T\right ) = 1\). The \(X_i\) are identically distributed and enter \(T\) symmetrically, so all \(n\) terms are equal, giving \[E\left (\frac {X_1}{T}\right ) = \frac {1}{n}.\]

Note. The expectation was obtained without integrating anything. Symmetry plus the fact that the ratios sum to one does all the work — a pattern worth remembering, because the direct calculation requires the Dirichlet distribution of \(U\) and is far longer.

Reference: Dennis D. Boss and Jacqueline M. Hughes - Oliver (19998).The American Statistician, 52(3), 218-221.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.