3.3 Estimating Equations
To find the M.L estimator, we usually solve the likelihood equation \[\sum ^n_{i = 1} S_i(\theta ,X_i) = 0.\] A function such as the score function which is a function of both the observations and the parameter is called an estimating function for example, if we know \(\var (X_i) = \theta \) independently identically distributed \(X_i\), then we can use the estimating function \[\Psi (\theta , X) = \sum ^n_{i = 1} \left (X_i - \overline {X}\right )^2 - (n - 1)\theta \] to estimate the parameter \(\theta \) without any knowledge of the distribution, density or mean of \(X\). The estimating function is the set equal to 0 and solved for \(\theta \).
Definition 3.3.1. An estimating function \(\Psi (\theta ,X)\) is unbiased if \(E\left [\Psi (\theta ,X)\right ] = 0\) for all \(\theta \).
Theorem 3.3.2. Suppose \(\widehat {\theta }\) is a root of the estimating equation \(\Psi (\theta ,X) = 0\), then \(\widehat {\theta }\) is approximately normal with mean \(\theta \) and variance \(\left [J(\Psi ,\theta )\right ]^{-1}\) where \[J(\Psi , \theta ) = \frac {\left [E\left (\frac {\partial \Psi }{\partial \theta }\right )\right ]^2}{\var \left [\psi (\theta ,X)\right ]}.\]
Note. \(J(\Psi , \theta )\) is called the Godambe information of the estimating function.
Example 3.3.3. Suppose \(X = (X_1, \, \cdots \, , \, X_n)\) is a random sample from a distribution with \(\, E\left [\log X_i\right ] = e^{\theta }\, \) and \(\, \var \left [\log X_i \right ] = e^{2\theta }\, , \, \, \, i = 1, \, 2, \, \cdots \, , \, n\).
Consider an estimating function
\[\Psi (\theta , X) = \sum ^n_{i = 1}\left (\log (X_i) - e^{\theta }\right )\]
- (a)
- Show that \(\Psi (\theta , X)\) is an unbiased estimating function.
- (b)
- Find the estimator \(\widehat {\theta }\) which satisfies \(\Phi \left (\widehat {\theta }, X\right ) = 0\).
- (c)
- Construct an approximate \(95\%\) C.I for \(\theta \).
Solution.
- (a)
- \(\begin {aligned}[t] E\left [\Psi (\theta , X)\right ] & = E\left [\sum ^n_{i = 1}\left (\log X_i - e^{\theta }\right )\right ] = \sum ^n_{i = 1} \left (E\left (\log X_i\right ) - e^{\theta }\right )\\ & = \sum ^n_{i = 1} \left (e^{\theta } - e^{\theta }\right )\\ & = \sum ^n_{i = 1} 0\\ & = 0. \end {aligned}\)
Therefore \(\, \Psi (\theta ,X)\) is an unbiased estimating function. - (b)
- \(\Psi (\theta ,X) = 0\) \[\sum ^n_{i = 1} \left (\log X_i - e^{\theta } \right ) = 0\] \[\sum ^n_{i = 1} \log X_i - n e^{\theta } = 0\] \[\sum ^n_{i = 1} \log X_i = ne^{\theta }\] \[\widehat {\theta } = \log \left [\frac {1}{n}\sum ^n_{i = 1} \log X_i\right ].\]
- (c)
- \(\widehat {\theta } \thicksim N\left (\theta , \left [J(\Psi , \theta )\right ]^{-1}\right )\) a \(95\%\) C.I for \(\theta \) is given by
\[\widehat {\theta } \pm 1.96 \sqrt {\dfrac {1}{J(\Psi ,\widehat {\theta }})}\]
\[J(\Psi ,\theta ) = \frac {E\left [E\left (\frac {\partial \Psi }{\partial \theta }\right )\right ]^2}{\var \left [\Psi (\theta ,X)\right ]}\] \[\Psi (\theta , X) = \sum ^n_{i = 1} \left (\log X_i - e^{\theta }\right )\] \[\frac {\partial \Psi }{\partial \theta } = - \sum ^n_{i = 1} e^{\theta } = -n e^{\theta }\] \[E\left (\frac {\partial \Psi }{\partial \theta }\right ) = -n e^{\theta }.\] \[\left [E\left (\frac {\partial \Psi }{\partial \theta }\right )\right ]^2 = n^2e^{2\theta }\]
\begin {align*} \var \left [\Psi (\theta ,X)\right ] & = \var \left [\sum ^n_{i = 1} \left (\log X_i - e^{\theta }\right )\right ] = \sum ^n_{i = 1} \var \left (\log X_i\right ) = \sum ^n_{i = 1} e^{2\theta } = ne^{2\theta }. \end {align*}
\[J(\Psi , \theta ) = \frac {n^2\, e^{2\theta }}{n\,e^{2\theta }} = n.\] \[J(\Psi ,\widehat {\theta }) = n\] \[\therefore \,\, \widehat {\theta } \, \pm \, 1.96\sqrt {\dfrac {1}{n}}\,.\]
Example 3.3.4. Suppose \(X_1, \, X_2, \, \cdots \, , \, X_n\) is a random sample from the Bernoulli\((\theta )\) distribution. Suppose that \(\varepsilon _1, \, \cdots \, , \, \varepsilon _n\) are independent \(N(0,\sigma ^2)\) random variables independent of the \(X_i's\). Define the estimating function \[\Psi \left [\theta , (X,Y)\right ] = \sum ^n_{i = 1}\left (Y_i - \theta \, X_i\right )\]
- (a)
- Show that this is an unbiased estimating function of \(\theta \).
- (b)
- Find the estimator \(\widehat {\theta }\) which satisfies \(\Psi \left [\theta , (X,Y)\right ] = 0\).
- (c)
- Construct an approximate \(95\%\) C.I for \(\theta \).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.