3.3 Estimating Equations
The ML estimator solves the likelihood equation \(\sum _{i=1}^n S_1(\theta ; X_i) = 0\). A function of the data and the parameter used in this way is called an estimating function, and it need not come from a likelihood. For example, if the \(X_i\) are independent and identically distributed with \(\var (X_i) = \theta \), the estimating function \[\Psi (\theta ; X) = \sum ^n_{i = 1}\left (X_i - \bar X\right )^2 - (n - 1)\theta \] can be set equal to zero and solved, giving \(\hat \theta = S^2\), without any knowledge of the form of the distribution or of its mean.
Definition 3.3.1. An estimating function \(\Psi (\theta ; X)\) is unbiased if \(E_\theta \left [\Psi (\theta ; X)\right ] = 0\) for all \(\theta \).
The score is an unbiased estimating function (theorem 2.2.1). The large-sample theory of section 2.5 carries over to roots of unbiased estimating equations, with a different information.
Theorem 3.3.2. Let \(\Psi (\theta ; X) = \sum _{i=1}^n\psi (\theta ; X_i)\) be an unbiased estimating function with \(X_1, \dots , X_n\) independent and identically distributed, and let \(\hat \theta \) be a consistent root of \(\Psi (\theta ; X) = 0\). Under regularity conditions \(\hat \theta \) is approximately normal with mean \(\theta \) and variance \(\frac {1}{J(\Psi , \theta )}\), where \[J(\Psi , \theta ) = \frac {\left [E\left (\frac {\partial \Psi }{\partial \theta }\right )\right ]^2}{\var \left [\Psi (\theta ; X)\right ]}\] is the Godambe information of the estimating function.
Sketch of proof. As in theorem 2.5.3, \(0 = \Psi (\hat \theta ) \approx \Psi (\theta ) + (\hat \theta - \theta )\frac {\partial \Psi }{\partial \theta }(\theta )\), so \(\hat \theta - \theta \approx -\Psi (\theta )\big /\frac {\partial \Psi }{\partial \theta }(\theta )\). The numerator is a sum of independent terms with mean zero and is approximately normal with variance \(\var (\Psi )\); the denominator is close to its expectation \(E\left (\frac {\partial \Psi }{\partial \theta }\right )\) by the law of large numbers. So \(\var (\hat \theta )\approx \var (\Psi )\big /\left [E\left (\frac {\partial \Psi }{\partial \theta }\right )\right ]^2\). □
For the score, \(E\left (\frac {\partial S}{\partial \theta }\right ) = -J(\theta )\) and \(\var (S) = J(\theta )\), so the Godambe information reduces to the Fisher information. Godambe showed that among unbiased estimating functions the score has the largest Godambe information, which is the estimating-equation version of the efficiency of maximum likelihood.
Example 3.3.3. Let \(X_1, \dots , X_n\) be a random sample from a distribution with \(E(\ln X_i) = e^\theta \) and \(\var (\ln X_i) = e^{2\theta }\). Consider \[\Psi (\theta ; X) = \sum ^n_{i = 1}\left (\ln X_i - e^\theta \right ).\]
- (a)
- Show that \(\Psi \) is unbiased.
- (b)
- Find the root \(\hat \theta \) of \(\Psi (\theta ; X) = 0\).
- (c)
- Construct an approximate \(95\%\) confidence interval for \(\theta \).
Solution.
- (a)
- \(E(\Psi ) = \sum \left [E(\ln X_i) - e^\theta \right ] = 0\).
- (b)
- \(\sum \ln X_i = ne^\theta \) gives \(\hat \theta = \ln \left (\frac 1n\sum \ln X_i\right )\), defined when \(\sum \ln X_i>0\), which holds with probability tending to one because the mean \(e^\theta \) is positive.
- (c)
- \(\frac {\partial \Psi }{\partial \theta } = -ne^\theta \), which is not random, and \(\var (\Psi ) = \sum \var (\ln X_i) = ne^{2\theta }\). So \[J(\Psi , \theta ) = \frac {n^2e^{2\theta }}{ne^{2\theta }} = n,\] and the approximate interval is \(\hat \theta \pm \frac {1.96}{\sqrt n}\).
Example 3.3.4. Let \(X_1, \dots , X_n\) be a random sample from the Bernoulli\((p)\) distribution with \(0<p<1\), and let \(Y_i = \theta X_i + \varepsilon _i\), where \(\varepsilon _1, \dots , \varepsilon _n\) are independent \(N(0, \sigma ^2)\) variables independent of the \(X_i\). Consider \[\Psi \left (\theta ; (X, Y)\right ) = \sum ^n_{i = 1}\left (Y_i - \theta X_i\right ).\]
- (a)
- Show that \(\Psi \) is an unbiased estimating function for \(\theta \).
- (b)
- Find the root \(\hat \theta \).
- (c)
- Construct an approximate \(95\%\) confidence interval for \(\theta \).
Solution.
- (a)
- \(Y_i - \theta X_i = \varepsilon _i\), so \(E(\Psi ) = \sum E(\varepsilon _i) = 0\) for every \(\theta \), whatever \(p\) and \(\sigma ^2\) are.
- (b)
- \(\sum Y_i = \theta \sum X_i\) gives \(\hat \theta = \frac {\sum Y_i}{\sum X_i}\), defined when at least one \(X_i = 1\).
- (c)
- \(\frac {\partial \Psi }{\partial \theta } = -\sum X_i\), with expectation \(-np\), and \(\var (\Psi ) = \var \left (\sum \varepsilon _i\right ) = n\sigma ^2\). So \[J(\Psi , \theta ) = \frac {n^2p^2}{n\sigma ^2} = \frac {np}{\sigma ^2} .\] Estimating \(p\) by \(\bar X\) and \(\sigma ^2\) by \(\hat \sigma ^2 = \frac {1}{n - 1}\sum \left (Y_i - \hat \theta X_i\right )^2\), the approximate interval is \[\hat \theta \pm 1.96\frac {\hat \sigma }{\sqrt {\sum X_i}} .\] Only the observations with \(X_i = 1\) carry information about \(\theta \), and the interval width depends on how many of them there are.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.