1.5 Completeness
The property of completeness is useful for determining the uniqueness of estimators and finding \(UMVUES\).
Definition 1.5.1. A statistic \(T\) or a family of distributions. \(T\) is complete if for every function \(h(x)\, , \, \hspace {0.1cm} E_{\theta }\left [h(T)\right ] = 0\,\), for all \(\theta \in \Omega \) implies \(\, P_{\theta }\left [h(T) = 0\right ] = 1,\,\,\) for all \(\theta \in \Omega \).
i.e \(\, \, E_{\theta }\left [h(T)\right ] = 0\, \) implies \(\, h(T) = 0\,\) with probability one.
Example 1.5.2. Let \(\, X_1\, , \, X_2\, , \, \cdots \cdots \, , \, X_n\,\) be a random sample from the \(N(\theta , 1)\) distribution. Consider \(T(X) = \left (X_1\, ,\, \sum ^n_{i = 1} X_i\right )\,\) prove that \(T(X)\) is sufficient for \(\, \{f_{\theta }(x)\, :\, \theta \in \Omega \}\) but not complete.
Solution. \begin {align*} f_{\theta }(x_1\,, \, x_2\, , \, \cdots \cdots \, , \, x_n)\, & = \, \prod ^n_{i = 1} \frac {1}{\sqrt {2\pi }}\, e^{\frac {-1}{2}(x_i - \theta )^2}\\\\ & = \, \frac {1}{\left (2\pi \right )^{n/2}}\, e^{\frac {-1}{2}\, \sum ^n_{i = 1}\left (x^2_i - 2\theta \, x_i + \theta ^2\right )}\\\\ & = \, \frac {1}{\left (2\pi \right )^{n/2}}\,\, e^{\frac {-1}{2}\left (\sum x_i^2 - 2\theta \sum x_i + n\theta ^2\right )}\\\\ & = \, \underbrace {e^{\frac {-1}{2}\left (n\theta ^2 - 2\theta \sum x_i\right )}}_{g\left (T(x);\, \theta \right )}\, \, \underbrace {\dfrac {1}{\left (2\pi \right )^{n/2}}\, e^{\frac {-1}{2}\sum x_i^2}}_{h(x)}\\\\ & = \, e^{\frac {-1}{2}\, \left [n\theta ^2 - 2\theta \left ( x_1 + \sum ^n_{i = 2} x_i\right )\right ]}\,\, \frac {1}{\left (2\pi \right )^{n/2}}e^{\frac {-1}{2}\, \sum x_i^2} \end {align*}
where \(\, \, T(X) \, = \, {\left (X_1\, , \, \sum ^n_{i = 1} X_i\right )}\).
\(\therefore \,\,\) By the Factorisation Theorem \(T(X)\) is a sufficient statistic.
Consider \(\,\, h(T) = X_1 - \overline {X}\). Then \begin {align*} E\left [h(T)\right ] & = E\left (X_1 - \overline {X}\right )\\ & = \, E\left (X_1\right ) \, - \, E\left (\overline {X}\right )\\ & = \, \theta \, -\, \theta \\ & = \, 0\hspace {0.7cm}\text {for}\hspace {0.3cm} \theta \in \Omega \end {align*}
\begin {align*} P_{\theta }\left (h(T) = 0\right ] & = P\left (X_1 - \overline {X} = 0\right )\\ & = 0\\ & \neq 1 \end {align*}
\(\therefore \,\, T\,\) is not complete. □
Example 1.5.3. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the Bernoulli\((\theta )\) distribution and let \(T(X) = \sum ^n_{i = 1} X_i\). Show that \(T(X)\) is complete and sufficient for \(\{f_{\theta }(x); \,\theta \in \Omega \}\).
Solution.
\[f_{\theta }(x_1,\, x_2, \, \cdots \, , \, x_n) = \prod ^n_{i = 1} \theta ^{x_i}\, \left (1 - \theta \right )^{1 - x_i} \, = \, \underbrace {\theta ^{\sum x_i}\, \left (1 - \theta \right )^{1 - \sum x_i}}_{g\left (T(X);\theta \right )}\, \underbrace {1}_{h(x)}.\]
Therefore by the factorization Theorem \(T(X) = \sum ^{n}_{i = 1}X_i\) is a sufficient statistic.
Now
\[T = \sum ^n_{i = 1}X_i\, \thicksim \, BIN(n,\theta )\]
suppose \(E_{\theta }\left [h(T)\right ] = 0\) for all \(\theta \). Then \begin {align*} E_{\theta }\left [h(T)\right ] & = \sum ^n_{t = 0} h(t)\, \binom {n}{t}\, \theta ^t\, \left (1 - \theta \right )^{n - t}\\ & = 0\\ & = \sum ^n_{t = 0} h(t)\, \binom {n}{t}\, \left (\frac {\theta }{1 - \theta }\right )^{t}\, \left (1 - \theta \right )^{n} = 0 \end {align*}
\[\sum ^n_{t = 0} h(t)\, \binom {n}{t}\, \left (\frac {\theta }{1 - \theta }\right )^t = 0\hspace {0.3cm} \text {since}\hspace {0.2cm} (1 - \theta )^n \neq 0, \, \, 0 < \theta < 1.\]
\[\sum ^n_{t = 0}h(t)\, \binom {n}{t}\, r^t = 0\, ;\hspace {0.3cm} \text {where}\hspace {0.2cm} r = \frac {\theta }{1 - \theta }\, , \, \,\, 0 < r < \infty .\]
This is a polynomial of degree \(n\) in \(r\) with coefficients \(h(t)\, \binom {n}{t}\) for \(t = 0, \, 1, \, \cdots \, , \, n\). For the polynomial to be 0, each
coefficient of \(r\) must be 0.
Since none of \(\binom {n}{t}\) terms is 0, this implies \(h(t) = 0, \, t = 0, \, 1, \, \cdots \, , \, n\) which means that \(P_{\theta }\left [h(T) = 0\right ] = 1\) for all \(\theta \in \Omega \). Therefore \(T\) is a complete
statistic. □
Problem 1.5.1. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(UNIF(0,\theta )\) distribution. Show that \(T(X) = X_{(n)}\) is a complete statistic for \(\{f_{\theta }(x);\, \theta \in \Omega \}\).
Show solution
Solution. The maximum has density \[f_T(t) = \frac {n\,t^{\,n-1}}{\theta ^{n}},\qquad 0<t<\theta .\] Suppose \(h\) is a function with \(E_{\theta }\left [h(T)\right ]=0\) for every \(\theta >0\). Then \[\frac {n}{\theta ^{n}}\int _0^{\theta } h(t)\,t^{\,n-1}\,dt = 0 \qquad \Longrightarrow \qquad \int _0^{\theta } h(t)\,t^{\,n-1}\,dt = 0 \quad \text {for every }\theta >0 ,\] since \(n/\theta ^{n}\neq 0\). The left-hand side is a function of its upper limit which vanishes identically, so differentiating with respect to \(\theta \) by the fundamental theorem of calculus gives \[h(\theta )\,\theta ^{\,n-1} = 0 \qquad \text {for almost every }\theta >0 .\] As \(\theta ^{n-1}\neq 0\) on \((0,\infty )\), this forces \(h(\theta )=0\) almost everywhere, which is exactly completeness.
Note. The whole argument is that the family of intervals \((0,\theta )\) grows to fill \((0,\infty )\) as \(\theta \) varies. Completeness here is a statement about the richness of the parameter set, not about the shape of the density — which is why restricting \(\theta \) to a finite set would destroy it.
Problem 1.5.2. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(N(\theta , a\theta ^2)\) distribution where \(a>0\) is a known constant and \(\theta >0\). Show that the minimal sufficient statistic is not complete.
Show solution
Solution.
The minimal sufficient statistic
Expanding the exponent, \[-\frac {1}{2a\theta ^{2}}\sum _{i=1}^{n}(x_i-\theta )^{2} = -\frac {1}{2a\theta ^{2}}\sum _{i=1}^{n}x_i^{2} + \frac {1}{a\theta }\sum _{i=1}^{n}x_i - \frac {n}{2a},\] so the density depends on the data only through \(T = \left (\sum _i X_i,\ \sum _i X_i^{2}\right )\), and the ratio criterion shows no further reduction is possible: the two coefficients \(1/(2a\theta ^{2})\) and \(1/(a\theta )\) move independently enough that matching the ratio for all \(\theta \) forces both components to agree. So \(T\) is minimal sufficient.
Why it is not complete
Both components can be scaled to estimate \(\theta ^{2}\) unbiasedly. Since \(E(X_i)=\theta \) and \(\var (X_i)=a\theta ^{2}\), \[E\left (\sum _i X_i^{2}\right ) = n\left (a\theta ^{2}+\theta ^{2}\right ) = n(1+a)\theta ^{2},\] \[E\left [\left (\sum _i X_i\right )^{2}\right ] = \var \left (\sum _i X_i\right ) + \left [E\left (\sum _i X_i\right )\right ]^{2} = na\theta ^{2} + n^{2}\theta ^{2} = n(a+n)\theta ^{2}.\] Hence the non-zero function \[h(T) = \frac {\sum _i X_i^{2}}{n(1+a)} - \frac {\left (\sum _i X_i\right )^{2}}{n(a+n)}\] satisfies \(E_{\theta }\left [h(T)\right ] = \theta ^{2}-\theta ^{2} = 0\) for every \(\theta >0\), while \(h\) is not identically zero. So \(T\) is not complete.
Remark. The model has one parameter but a two-dimensional minimal sufficient statistic, and that mismatch is the whole reason completeness fails: there are two separate routes to \(\theta ^{2}\) and their difference is a non-trivial function of \(T\) with zero mean. It also means the Lehmann–Scheffé theorem is unavailable here, so an unbiased estimator based on \(T\) carries no guarantee of being the UMVUE.
1.5.1 Completeness, Sufficiency and Minimality
Theorem 1.5.4. If \(T(X)\) is a complete and sufficient statistic for the model \(\{f_{\theta }(x);\, \theta \in \Omega \}\), then \(T(X)\) is a minimal sufficient statistic.
Proof. Assume \(E_{\theta }\left |T\right |<\infty \), and let \(S=S(X)\) be a minimal sufficient statistic for the model. Since \(T\) is sufficient and \(S\) is minimal, \(S\) is a function of \(T\): there is a measurable \(r\) with \(S=r(T)\).
Because \(S\) is sufficient, the conditional expectation \[h(S) := E_{\theta }\left (T \mid S\right )\] does not depend on \(\theta \), so \(h\) is a genuine statistic. By the law of total expectation, for every \(\theta \in \Omega \), \[E_{\theta }\left [\,T - h(r(T))\,\right ] = E_{\theta }(T) - E_{\theta }\left [E_{\theta }\left (T\mid S\right )\right ] = E_{\theta }(T) - E_{\theta }(T) = 0 .\] The quantity in brackets is a function of \(T\) alone with zero expectation for every \(\theta \). Completeness of \(T\) therefore forces \[T = h(r(T)) = h(S) \qquad \text {with probability } 1 .\] So \(T\) is a function of \(S\), while \(S=r(T)\) is a function of \(T\). The two statistics are equivalent, and since \(S\) is minimal sufficient, so is \(T\). □
Note. This is Bahadur’s theorem, and it is the reason completeness is worth checking: it upgrades a sufficient statistic to a minimal one without any of the work of comparing likelihood ratios. The converse fails — minimal sufficiency does not imply completeness — and the problem immediately below constructs the standard counterexample.
Problem 1.5.3. The converse of the above theorem is not true. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(UNIF(\theta - 1, \theta + 1)\) distribution. Show that \(T(X) = \left (X_{(1)}, X_{(n)}\right )\) is minimal sufficient. Show also that for the nonzero function \[h(T) = \frac {X_{(n)} - X_{(1)}}{2} - \frac {n - 1}{n + 1}\] \(E_{\theta }\left [h(T)\right ] =0\) for all \(\theta \in \Omega \) and therefore \(T\) is not a complete statistic.
Show solution
Solution.
Minimal sufficiency
The joint density is one exactly when every observation lies in \((\theta -1,\theta +1)\): \[f_{\theta }(x) = \prod _{i=1}^{n} \frac {1}{2}\,I\left (\theta -1<x_i<\theta +1\right ) = 2^{-n}\,I\left (x_{(n)}-1<\theta <x_{(1)}+1\right ).\] For two samples the ratio \(f_{\theta }(x)/f_{\theta }(y)\) is a constant function of \(\theta \) precisely when the intervals \(\left (x_{(n)}-1,\ x_{(1)}+1\right )\) and \(\left (y_{(n)}-1,\ y_{(1)}+1\right )\) coincide, that is when \(x_{(1)}=y_{(1)}\) and \(x_{(n)}=y_{(n)}\). So \(T=\left (X_{(1)},X_{(n)}\right )\) is minimal sufficient.
It is not complete
Put \(R = X_{(n)}-X_{(1)}\), the sample range. The observations are uniform on an interval of length \(2\), and the range of \(n\) independent uniforms on an interval of length \(L\) has mean \(L(n-1)/(n+1)\). With \(L=2\), \[E_{\theta }(R) = \frac {2(n-1)}{n+1}\qquad \text {for every }\theta ,\] the value being free of \(\theta \) because the range is invariant under the shift \(x\mapsto x+c\) that carries one parameter value to another. Therefore \[E_{\theta }\left [h(T)\right ] = E_{\theta }\left [\frac {X_{(n)}-X_{(1)}}{2}\right ] - \frac {n-1}{n+1} = \frac {n-1}{n+1} - \frac {n-1}{n+1} = 0 \qquad \text {for all }\theta \in \Omega ,\] while \(h\) is plainly not the zero function — the range is genuinely random. So \(T\) is minimal sufficient but not complete, which is the promised counterexample to the converse.
Note. The mechanism is worth naming. The range is an ancillary statistic: its distribution does not involve \(\theta \) at all. Any non-constant ancillary function of a statistic destroys completeness immediately, by centring it. This is the same phenomenon Basu’s theorem exploits from the other direction.
Problem 1.5.4. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(UNIF(0,\theta )\) distribution. prove that \(T(X) = X_{(n)}\) is a minimal sufficient statistic for this model.
Show solution
Solution. For the UNIF\((0,\theta )\) model the joint density is \[f_{\theta }(x) = \frac {1}{\theta ^{n}}\prod _{i=1}^{n}I\left (0<x_i<\theta \right ) = \frac {1}{\theta ^{n}}\,I\left (x_{(1)}>0\right )\,I\left (x_{(n)}<\theta \right ).\] Sufficiency of \(X_{(n)}\) follows from the factorisation criterion at once, with \(g\left (x_{(n)},\theta \right )=\theta ^{-n}I\left (x_{(n)}<\theta \right )\) and \(h(x)=I\left (x_{(1)}>0\right )\).
For minimality, take two samples in the support and form \[\frac {f_{\theta }(x)}{f_{\theta }(y)} = \frac {I\left (x_{(n)}<\theta \right )}{I\left (y_{(n)}<\theta \right )} .\] Viewed as a function of \(\theta \), this is constant only if the two indicators change value at the same point, that is only if \(x_{(n)} = y_{(n)}\). Hence \(X_{(n)}\) is minimal sufficient.
Combined with the completeness shown above, \(X_{(n)}\) is complete and sufficient, so by Bahadur’s theorem its minimality also follows without this calculation — but the direct argument is shorter than the theorem it instantiates.
Problem 1.5.5. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(EXP(1, \theta )\) distribution. Show that \(T(X) = X_{(1)}\) is a minimal sufficient statistic for the model.
Show solution
Solution. This is the calculation already carried out in the previous section, and it is worth recording that the two are the same problem. The joint density is \[f_{\theta }(x) = e^{n\theta }\,I\left (x_{(1)}>\theta \right )\cdot e^{-\sum _i x_i},\] so \(X_{(1)}\) is sufficient by factorisation, and the ratio \[\frac {f_{\theta }(x)}{f_{\theta }(y)} = e^{\sum _i y_i-\sum _i x_i}\cdot \frac {I\left (x_{(1)}>\theta \right )}{I\left (y_{(1)}>\theta \right )}\] is free of \(\theta \) only when the two indicators switch at the same place, that is only when \(x_{(1)}=y_{(1)}\). Hence \(X_{(1)}\) is minimal sufficient.
Completeness holds as well, by the same upper-limit differentiation used for the UNIF\((0,\theta )\) maximum: \(X_{(1)}\) has density \(n\,e^{-n(t-\theta )}\) on \((\theta ,\infty )\), and \(\int _{\theta }^{\infty }h(t)e^{-nt}\,dt = 0\) for every \(\theta \) forces \(h=0\) almost everywhere. So \(X_{(1)}\) is complete and sufficient, which is what the next problem needs.
1.5.2 Conditioning: Total Expectation and Total Variance
Theorem 1.5.5. For any random variables \(X\) and \(Y,\,\) \(E_{\theta }(X) = E_{\theta }\left [E_{\theta }(X \mid Y)\right ]\) and
\(\var (X) = E_{\theta }\left [\var _{\theta }(X \mid Y)\right ] \, + \, \var _{\theta }\left [E_{\theta }(X \mid Y)\right ].\)
Proof.
The first identity
Writing the inner expectation out and exchanging the order of integration, \[E\left [E(X\mid Y)\right ] = \int \left [\int x\, f_{X\mid Y}(x\mid y)\,dx\right ] f_Y(y)\,dy = \iint x\, f_{X\mid Y}(x\mid y)\, f_Y(y)\,dx\,dy .\] The product \(f_{X\mid Y}(x\mid y)f_Y(y)\) is the joint density \(f(x,y)\), so the double integral is \(\iint x\,f(x,y)\,dx\,dy = E(X)\). In the discrete case replace the integrals by sums; the argument is unchanged.
The second identity
Apply the first identity to \(X^{2}\) and use \(\var (X\mid Y) = E\left (X^{2}\mid Y\right ) - \left [E(X\mid Y)\right ]^{2}\): \begin {align*} E\left (X^{2}\right ) &= E\left [E\left (X^{2}\mid Y\right )\right ] = E\left [\var (X\mid Y) + \left \{E(X\mid Y)\right \}^{2}\right ]\\ &= E\left [\var (X\mid Y)\right ] + E\left [\left \{E(X\mid Y)\right \}^{2}\right ]. \end {align*}
Subtracting \(\left [E(X)\right ]^{2} = \left \{E\left [E(X\mid Y)\right ]\right \}^{2}\) from both sides, \begin {align*} \var (X) &= E\left (X^{2}\right ) - \left [E(X)\right ]^{2}\\ &= E\left [\var (X\mid Y)\right ] + \underbrace {E\left [\left \{E(X\mid Y)\right \}^{2}\right ] - \left \{E\left [E(X\mid Y)\right ]\right \}^{2}}_{\textstyle =\ \var \left [E(X\mid Y)\right ]}, \end {align*}
which is the stated decomposition. □
Remark. The second identity is the whole content of the Rao–Blackwell theorem below. Conditioning on a sufficient statistic replaces an estimator by its conditional mean, and the decomposition says the variance then drops by exactly \(E\left [\var (X\mid Y)\right ]\) — a quantity that is zero only when \(X\) was already a function of \(Y\).
Theorem 1.5.6. If \(T(X)\) is a complete statistic, then there is at most one function of \(T\) that provides an unbiased estimator of the parameter \(\tau (\theta )\).
Proof. Suppose \(g_1(T)\) and \(g_2(T)\) are both unbiased for \(\tau (\theta )\), so that \[E_{\theta }\left [g_1(T)\right ] = E_{\theta }\left [g_2(T)\right ] = \tau (\theta ) \qquad \text {for every } \theta \in \Omega .\] Put \(h(T) = g_1(T) - g_2(T)\). Then \(h\) is a function of \(T\) alone and \[E_{\theta }\left [h(T)\right ] = \tau (\theta ) - \tau (\theta ) = 0 \qquad \text {for every } \theta \in \Omega .\] Completeness of \(T\) says precisely that the only such function is the zero function up to a null set: \(P_{\theta }\left (h(T)=0\right )=1\) for every \(\theta \). Hence \(g_1(T) = g_2(T)\) with probability one, and the two estimators are the same. □
1.5.3 Minimum Variance Unbiased Estimation
Theorem 1.5.7. If \(T(X)\) is a complete sufficient statistic for the model \(\{f_{\theta }(x);\, \theta \in \Omega \}\) and \(E_{\theta }\left [g(T)\right ] = \tau (\theta )\), then \(g(T)\) is the \(UMVUE\) of \(\tau (\theta )\).
Proof. Since \(T(X)\) is a complete sufficient statistic by Theorem 1.5.11, \(g(T)\) is unique. Suppose \(U = U(X)\) is an unbiased estimator of \(\tau (\theta )\). Let \(g_1(T) = E\left (U \mid T\right )\) (doesn’t depend on \(\theta \)) \begin {align*} E_{\theta }\left [g_1(T)\right ] & = E_{\theta }\left [E\left (U \mid T\right )\right ]\\ & = E(U)\\ & = \tau (\theta ). \end {align*}
By Theorem 1.5.11 \(g_1(T) = g(T)\). Now \begin {align*} \var _{\theta }(U) & = \underbrace {E_{\theta }\left [\var (U \mid T)\right ]}_{\geq 0}\, + \, \var _{\theta }\Big [\underbrace {E(U \mid T)}_{g(T)}\Big ]\\ & \geq \var \left [g(T)\right ] \end {align*}
i.e \(\, \var _{\theta }(U) \geq \var \left [g(T)\right ]\). Therefore \(g(T)\) is the \(UMVUE\). □
Example 1.5.8. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the Bernoulli\((\theta )\) distribution. Find the \(UMVUE\) of \(\theta ^2\).
Solution. We already know that \(T(X) = \sum ^n_{i = 1} X_i\) is a complete sufficient statistic.
Consider \(g(T) = \overline {X}\). Then \(E(\overline {X}) = \theta \)
\[\var \left (\overline {X}\right ) = E\left (\overline {X}^2\right ) - \left (E(\overline {X})\right )^2\]
\[\implies \,\, \frac {\var (X)}{n} = E\left (\overline {X}^2\right ) - \theta ^2\]
\[\frac {\theta (1 - \theta )}{n} + \theta ^2 = E\left (\overline {X}^2\right )\, \implies \, \frac {\theta - \theta ^2 + n\theta ^2}{n} = E\left (\overline {X}^2\right )\]
\[\implies \,\, \theta + (n - 1)\theta ^2 = n \, E\left (\overline {X}^2\right )\]
\[(n - 1)\theta ^2 = n\, E\left (\overline {X}^2\right ) - \theta \]
\[n\, E\left (\overline {X}^2\right ) - E(\overline {X}) = (n - 1)\, \theta ^2\]
\[\implies \,\, \frac {E\left [n\, \overline {X} - \overline {X}\right ]}{n - 1} = \theta ^2\]
\[\implies \,\, E\left [\frac {\overline {X}(n\, \overline {X} - 1)}{n - 1}\right ] = \theta ^2\]
Therefore \(\, \dfrac {\overline {X}(n \, \overline {X} - 1)}{n - 1}\,\) is the \(UMVUE\) of \(\theta ^2\).
or \[E\left [\frac {\sum X_i\, \left (\sum X_i - 1\right )}{n(n - 1)}\right ] = \theta ^2\]
\[E\left [\frac {T(T - 1)}{n (n - 1)}\right ] = \theta ^2\] Therefore \(\, \dfrac {T (T - 1)}{n (n - 1)}\,\) is the \(UMVUE\) of \(\theta ^2\). □
Example 1.5.9. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(UNIF(0,\theta )\) distribution. Find the \(UMVUE\) of \(\tau (\theta ) = \theta \).
Solution. \(T = X_{(n)}\) is a complete sufficient statistic (Problem 1.5.4). By Example 1.2.12 \[E(T) = \frac {n}{n + 1}\, \theta \,\, \implies \,\, \frac {n + 1}{n}\, E(T) = \theta \] \[\implies \,\, E\left (\frac {(n + 1)}{n}\, T\right ) = \theta .\] Therefore \(\, \dfrac {n + 1}{n}\, T\,\) is the \(UMVUE\) of \(\tau (\theta ) = \theta \). □
Problem 1.5.6. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the Bernoulli\((\theta )\) distribution. Find the \(UMVUE\) of \(\tau (\theta ) = \theta (1 -\theta )\).
Show solution
Solution. \(T=\sum _{i=1}^{n}X_i\sim \) BIN\((n,\theta )\) is complete and sufficient, so by Lehmann–Scheffé it is enough to find any function of \(T\) that is unbiased for \(\theta (1-\theta )\); it is then automatically the UMVUE.
Try \(T(n-T)\). Using \(E(T)=n\theta \) and \(E\left (T^{2}\right )=n\theta (1-\theta )+n^{2}\theta ^{2}\), \begin {align*} E\left [T(n-T)\right ] &= nE(T) - E\left (T^{2}\right )\\ &= n^{2}\theta - n\theta (1-\theta ) - n^{2}\theta ^{2}\\ &= n^{2}\theta (1-\theta ) - n\theta (1-\theta ) = n(n-1)\,\theta (1-\theta ). \end {align*}
Dividing by the constant \(n(n-1)\), \[\widehat {\tau } = \frac {T(n-T)}{n(n-1)} = \frac {n}{n-1}\,\overline {X}\left (1-\overline {X}\right )\] is unbiased, and therefore is the UMVUE of \(\theta (1-\theta )\).
Note. The naive estimator \(\overline {X}\left (1-\overline {X}\right )\) is biased downwards by exactly the factor \((n-1)/n\). The correction is the same one that turns the sample variance into its unbiased form, and for the same reason: a product of an estimator with itself underestimates the product of the parameters, because it inherits the estimator’s own variability.
Problem 1.5.7. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(EXP(1,\theta )\) distribution. Find the \(UMVUE\) of \(\tau (\theta ) = \theta ^2\).
Show solution
Solution. \(T=X_{(1)}\) is complete and sufficient for this model, as shown above. Its density is \(n\,e^{-n(t-\theta )}\) on \((\theta ,\infty )\), so \(T-\theta \) is exponential with rate \(n\) and \[E(T) = \theta + \frac {1}{n},\qquad \var (T) = \frac {1}{n^{2}},\qquad E\left (T^{2}\right ) = \frac {1}{n^{2}} + \left (\theta +\frac 1n\right )^{2} = \theta ^{2} + \frac {2\theta }{n} + \frac {2}{n^{2}} .\] Look for an unbiased estimator of the form \(g(T)=T^{2}+\alpha T+\beta \): \[E\left [g(T)\right ] = \theta ^{2} + \left (\frac {2}{n}+\alpha \right )\theta + \left (\frac {2}{n^{2}} + \frac {\alpha }{n} + \beta \right ).\] Matching \(\theta ^{2}\) requires the coefficient of \(\theta \) and the constant to vanish, giving \(\alpha =-2/n\) and then \(\beta =0\). Hence \[\widehat {\theta ^{2}} = X_{(1)}^{2} - \frac {2X_{(1)}}{n},\] and by Lehmann–Scheffé this is the UMVUE of \(\theta ^{2}\).
Remark. Note the estimator can be negative — it is negative whenever \(0<X_{(1)}<2/n\) — even though \(\theta ^{2}\geq 0\) always. This is not an error. Unbiasedness is a statement about an average over repeated samples and carries no promise that individual values land in the parameter space, and the UMVUE can be inadmissible for exactly this reason. Truncating at zero would reduce the mean square error while destroying unbiasedness.
1.5.4 The Lehmann–Scheffé Theorem
Theorem 1.5.10 (Lehmman - Scheffe’). If \(T(X)\) is a complete sufficient statistic for the model \(\{f_{\theta }(x):\, \theta \in \Omega \}\) and \(U(X)\) is any unbiased estimator of \(\tau (\theta )\), then \(E\left (U \mid T\right )\) is the \(UMVUE\) of \(\tau (\theta )\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.