1.4 Minimal Sufficiency
Recall that the purpose of a sufficient statistic is to achieve data reduction without loss of information about the parameter \(\theta \). Since there are many sufficient statistics in a problem, we prefer one that achieves most data reduction such that a statistic is a minimal sufficient statistic.
Example 1.4.1. A sufficient statistic \(T(X)\) is called a minimal sufficient statistic for \(\, \{f_{\theta }(x)\, :\, \theta \in \Omega \}\,\) if for any
other sufficient statistic \(U(X)\), there exists a function \(g(\cdot )\) such that \(\, T(X) = g\left (U(X)\right )\).
i.e A minimal sufficient statistic can be recovered from any other sufficient statistic.
Theorem 1.4.2. Suppose there exists a function \(T(X)\) such that for the model \(\, \{f_{\theta }(x)\, :\, \theta \in \Omega \}\,\) an for any two sample points \(x\) and \(y\), the ratio \(\, \dfrac {f_{\theta }(x)}{f_{\theta }(y)}\,\) is constant as a function of \(\theta \) if and only if \(\, T(x) = T(y)\). Then \(T(X)\) is a minimal sufficient statistic for \(\theta \).
In each of the previous four example \(POI(\theta )\, , \, \, N(\mu , \sigma ^2)\, , \, \, WEI(1,\theta )\, \) and \(\, UNIF(0, \theta )\). We found a minimal sufficient statistic as it is not possible to find a further reduction of the data and still maintain the property of sufficiency.
Example 1.4.3. Let \(\, X_1\, , \, X_2\, , \, \cdots \cdots \, , \, X_n\,\) be a random sample from a distribution with pdf \[f_{\theta }(x) \, =\, \theta \, x^{\theta - 1}\hspace {0.5cm}, \hspace {0.5cm} 0 < x < 1\hspace {0.5cm}, \hspace {0.5cm} \theta > 0\] Find a minimal sufficient statistic for \(\, \{f_{\theta }(x)\, :\, \theta \in \Omega \}\)
Solution. \begin {align*} f_{\theta }(x_1\, ,\, \cdots \cdots \, , \, x_n) \, & = \, \prod ^n_{i = 1}\, \theta \, x_i^{\theta - 1}\, = \,\theta ^n\, \left (\prod ^n_{i = 1} x_i \right )^{\theta - 1}. \end {align*}
Let \(x = (x_1\, , \, \cdots \cdots \, x_n)\, \) and \(\, y = (y_1\, , \, \cdots \cdots \, , \, y_n)\,\) be two possible outcomes, then \begin {align*} \frac {f_{\theta }(x)}{f_{\theta }(y)} = \frac {\theta ^n\, \left (\prod ^n_{i = 1} x_i\right )^{\theta - 1}}{\theta ^n\, \left (\prod ^n_{i = 1}y_i \right )^{\theta - 1}}\, = \,\left ( \frac {\prod ^n_{i = i}\, x_i}{\prod ^n_{i = 1}y_i}\right )^{\theta - 1} \end {align*}
This ratio does not depend on \(\theta \) if and only if (iff) \(\,\prod ^n_{i = 1}\, x_i \, = \, \prod ^n_{i = 1} \, y_i\,\), therefore \(\, T(X) = \prod ^n_{i = 1}\, x_i\,\) is a minimal sufficient statistic for this model. □
Example 1.4.4. Let \(\, X_1\, , \, X_2\, , \, \cdots \cdots \, ,\, X_n\, \) be a random sample from the \(N(\theta , \theta ^2)\) distribution. Find a minimal sufficient statistic for \(\, \{f_{\theta }(x)\, :\, \theta \in \Omega \}\).
Solution. \begin {align*} f(x_1\, , \, x_2\, , \, \cdots \cdots \, , \, x_n)& = \, \prod ^n_{i = 1} \frac {1}{\sqrt {2\pi \theta ^2\,}}\, e^{\frac {-1}{2\theta ^2}(x_i - \theta )^2}\\\\ & = \, \frac {1}{\left (2\pi \, \theta ^2\,\right )^{n/2}}\, e^{\frac {-1}{2\theta ^2}\, \sum ^n_{i = 1}(x_i - \theta )^2}\\\\ & = \, \frac {1}{\left (2\pi \theta ^2\right )^{n/2}}\, e^{\frac {-1}{2\theta ^2}\, \left (\sum x_i^2 - 2\theta \sum x_i + n\theta ^2\right )} \end {align*}
Let \(\, x = (x_1\, , \, x_2\, , \, \cdots \cdots \, , \, x_n)\,\) and \(y = (y_1\, ,\, y_2\, , \, \cdots \cdots \, , \, y_n)\,\) be points, then \begin {align*} \frac {f_{\theta }(x)}{f_{\theta }(y)} \, & = \, \frac {\dfrac {1}{\left (2\pi \theta ^2\right )^{n/2}}\, e^{\frac {-1}{2\theta ^2}\, \left (\sum x_i^2 - 2\theta \, \sum x_i + n\theta ^2 \right )}}{\dfrac {1}{\left (2\pi \theta ^2\right )^{n/2}}\, e^{\frac {-1}{2\theta ^2}\left (\sum y_i^2 - 2\theta \sum y_i + n\theta ^2\right )}}\\\\ & =\, e^{\frac {-1}{2\theta ^2}\, \left (\sum x_i^2 - \sum y_i^2 - 2\theta \left (\sum x_i - \sum y_i\right )\right )} \end {align*}
This ratio does not depend on \(\theta \) if and only if \(\, \sum ^n_{i = 1}\, x^2_i \, = \, \sum ^n_{i = 1}\, y^2_i\,\) and \(\, \sum ^n_{i = 1} \, x_i \, = \, \sum ^n_{i = 1}\, y_i\)
\(\therefore \,\, T(X) = {\left (\sum ^n_{i = 1} X_i \, , \, \sum ^n_{i = 1}X^2_i\right )}\,\,\) is a minimal sufficient statistic for this model. □
Example 1.4.5. Let \(\, X_1\, , \, X_2\, , \, \cdots \cdots \, , \, X_n\,\) be a random sample from the \(UNIF(0,\theta )\) distribution. Find a minimal sufficient statistic for \(\, \{f_{\theta }(x):\, \theta \in \Omega \}\,\).
Solution. \begin {align*} f_{\theta }(x_1\, , \, x_2\, , \, \cdots \cdots \, , \, x_n)\, & = \, \frac {1}{\theta ^n}\hspace {1cm},\hspace {0.5cm} 0 < x_1\,, \cdots \cdots \, x_n < \theta \\\\ & = \, \frac {1}{\theta ^n}\hspace {1cm}, \hspace {0.5cm} 0 < x_{(1)}\, , \, x_{(n)} < \theta \\\\ & = \, \frac {1}{\theta ^n}\, I\left (x_{(1)} > 0\right )\, I\left (x_{(n)} < \theta \right ) \end {align*}
Let \(\, x = (x_1\, , \, x_2\, , \, \cdots \cdots \, ,\, x_n)\,\) and \(\, y = (y_1\, , \, y_2\, , \, \cdots \cdots \, , \, y_n)\,\) be two possible outcomes, then \begin {align*} \frac {f_{\theta }(x)}{f_{\theta }(y)} \, & = \, \frac {\dfrac {1}{\theta ^n}\, I\left (x_{(1)} > 0\right )\, I\left (x_{(n)} < \theta \right )}{\dfrac {1}{\theta ^n}\, I\left (y_{(1)} > 0\right )\, I\left (y_{(n)}< \theta \right )}\\\\ & = \, \frac {I\left (x_{(1)} > 0\right )\, I\left (x_{(n)} < \theta \right )}{I\left (y_{(1)} > 0\right )\, I\left (y_{(n)} < \theta \right )} \end {align*}
If \(\, x_{(n)} = y_{(n)}\,\) then the ratio does not depend on \(\theta \).
Suppose \(\, x_{(n)} > y_{(n)}\).
If \(\theta \) is such that \(\theta > x_{(n)} > y_{(n)}\, , \, \) then
\[\dfrac {f_{\theta }(x)}{f_{\theta }(y)}\, = \, \frac {I\left (x_{(1)} > 0\right )}{I\left (y_{(1)} > 0\right )}\]
If \(\theta \) is such that \(x_{(n)} > \theta > y_{(n)}\,\) \[\frac {f_{\theta }(x)}{f_{\theta }(y)}\, = \, 0\] Therefore the value of \(\, \dfrac {f_{\theta }(x)}{f_{\theta }(y)}\,\) depends on \(\theta \) if \(x_{(n)} > y_{(n)}\).
A similar argument holds for the case \(\, x_{(n)} < y_{(n)}\).
\(\therefore \,\, \) the ratio does not depend on \(\theta \) if and only if \(x_{(n)} = y_{(n)}\).
Therefore \(\,\, T(X) = X_{(n)}\,\) is a minimal sufficient statistics for the model. □
Problem 1.4.1. Let \(\, X_1\, , \, X_2\, , \, \cdots \cdots \, , \, X_n\,\) be a random sample from the \(UNIF(\theta , \theta + 1)\,\) distribution. Find a minimal sufficient statistic for \(\, \{f_{\theta }(x)\, :\, \theta \in \Omega \}\).
Show solution
Solution. The joint density is one exactly when every observation lies in \((\theta ,\theta +1)\), that is \[f_{\theta }(x) = \prod _{i=1}^{n} I\left (\theta <x_i<\theta +1\right ) = I\left (x_{(n)}-1 < \theta < x_{(1)}\right ),\] and zero otherwise. Sufficiency of \(T=\left (X_{(1)},X_{(n)}\right )\) is immediate from this display.
For minimality use the ratio criterion. For two samples \(x\) and \(y\) in the support, \[\frac {f_{\theta }(x)}{f_{\theta }(y)} = \frac {I\left (x_{(n)}-1<\theta <x_{(1)}\right )}{I\left (y_{(n)}-1<\theta <y_{(1)}\right )} .\] This ratio is a constant function of \(\theta \) precisely when the two intervals \(\left (x_{(n)}-1,\ x_{(1)}\right )\) and \(\left (y_{(n)}-1,\ y_{(1)}\right )\) coincide, and two intervals coincide exactly when their endpoints agree, that is when \(x_{(1)}=y_{(1)}\) and \(x_{(n)}=y_{(n)}\). Hence \[T(X) = \left (X_{(1)},\, X_{(n)}\right )\] is minimal sufficient.
Remark. The parameter is one-dimensional and the minimal sufficient statistic is two-dimensional. There is no contradiction — minimal means no further reduction is possible, not that the dimension matches. It also shows why this model is a standard source of counterexamples: the same \(T\) is minimal sufficient but, as the next problem in the completeness section shows, it is not complete.
Problem 1.4.2. Let \(\, X_1\, , \, X_2\, , \, \cdots \cdots \, , \, X_n\,\) be a random sample from the \(EXP(1, \theta )\) distribution. Show that \(X_{(1)}\) is sufficient statistic for the model and find the conditional distribution for \(\, X_1\, , \, X_2\, , \, \cdots \cdots \, , \, X_n\,\) given \(X_{(1)} = t\). Prove that \(X_{(1)}\) is a minimal sufficient statistic.
Show solution
Solution.
Sufficiency
As in the previous section, the joint density factorises as \[f_{\theta }(x) = e^{n\theta }\,I\left (x_{(1)}>\theta \right )\cdot e^{-\sum _i x_i},\] so \(X_{(1)}\) is sufficient.
The conditional distribution
The density of \(X_{(1)}\) is that of the minimum of \(n\) independent shifted exponentials, \[f_{X_{(1)}}(t) = n\,e^{-n(t-\theta )},\qquad t>\theta ,\] so on the set where \(\min _i x_i = t\), \[f\left (x \mid X_{(1)}=t\right ) = \frac {e^{n\theta }e^{-\sum _i x_i}}{n\,e^{-n(t-\theta )}} = \frac {e^{nt-\sum _i x_i}}{n} .\] The result does not involve \(\theta \), which confirms sufficiency directly rather than through the factorisation criterion. Read another way: given \(X_{(1)}=t\), the label of the minimising index is uniform on \(\{1,\dots ,n\}\) and the remaining \(n-1\) observations are distributed as \(t\) plus independent EXP\((1,0)\) variables — the memoryless property, with the threshold reset to \(t\).
Minimality
For two samples, \[\frac {f_{\theta }(x)}{f_{\theta }(y)} = e^{\sum _i y_i - \sum _i x_i}\cdot \frac {I\left (x_{(1)}>\theta \right )}{I\left (y_{(1)}>\theta \right )} .\] The first factor is free of \(\theta \) already. The second is a constant function of \(\theta \) only if the two indicators switch at the same place, that is only if \(x_{(1)} = y_{(1)}\). Hence \(X_{(1)}\) is minimal sufficient.
Problem 1.4.3. Let \(\, X_1\, , \, X_2\, , \, \cdots \cdots \, , \, X_n\,\) be a random sample from the \(CAU(1,\theta )\) distribution. Find a minimal sufficient for \(\, \{f_{\theta }(x)\, :\, \theta \in \Omega \}\).
Show solution
Solution. The Cauchy density with location \(\theta \) and scale \(1\) is \(f_{\theta }(x) = \left [\pi \left (1+(x-\theta )^{2}\right )\right ]^{-1}\), so \[f_{\theta }(x_1,\dots ,x_n) = \frac {1}{\pi ^{n}}\prod _{i=1}^{n}\frac {1}{1+(x_i-\theta )^{2}} .\] For two samples \(x\) and \(y\) the ratio is \[\frac {f_{\theta }(x)}{f_{\theta }(y)} = \frac {\prod _{i=1}^{n}\left [1+(y_i-\theta )^{2}\right ]} {\prod _{i=1}^{n}\left [1+(x_i-\theta )^{2}\right ]} .\] Regard numerator and denominator as polynomials in \(\theta \). Each is monic of degree \(2n\). The ratio is a constant function of \(\theta \) exactly when the two polynomials are proportional, and two monic polynomials of the same degree are proportional only if they are equal. Equal polynomials have the same roots with the same multiplicities; the roots of \(\prod _i\left [1+(x_i-\theta )^{2}\right ]\) are \(\theta = x_i \pm i\), so the multiset \(\{x_1,\dots ,x_n\}\) is determined. Hence the ratio is free of \(\theta \) if and only if \(y\) is a permutation of \(x\), and the minimal sufficient statistic is the full set of order statistics \[T(X) = \left (X_{(1)},\, X_{(2)},\, \dots ,\, X_{(n)}\right ).\]
Remark. No reduction of the data is possible beyond discarding the order in which the observations arrived. This is the standard warning against expecting a low-dimensional summary: the Cauchy location model has one parameter and a minimal sufficient statistic of dimension \(n\). It is also why the sample mean is useless here — \(\overline {X}\) has the same Cauchy distribution as a single observation, so averaging buys nothing at all.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.