Proof.

(i)

For one observation, \[E_\theta \left [S_1(\theta ; X)\right ] = \int _A\frac {f'_\theta (x)}{f_\theta (x)}f_\theta (x)\,dx = \int _A f'_\theta (x)\,dx = 0 ,\] and \(S\) is a sum of \(n\) such terms.

(ii)

By (i), \(\var (S) = E(S^2)\). Since \(S = \sum _i S_1(\theta ; X_i)\) with independent terms of mean zero, the cross terms \(E\left [S_1(\theta ; X_i)S_1(\theta ; X_j)\right ] = 0\) for \(i\ne j\), and \[E(S^2) = \sum _{i=1}^n E\left [S_1(\theta ; X_i)^2\right ] = \sum _{i=1}^n E\left [\left (\frac {f'_\theta (X_i)}{f_\theta (X_i)}\right )^2\right ].\] On the other hand, taking expectations in (2.1), \[J(\theta ) = \sum _{i=1}^n E\left [\left (\frac {f'_\theta (X_i)}{f_\theta (X_i)}\right )^2\right ] - \sum _{i=1}^n E\left [\frac {f''_\theta (X_i)}{f_\theta (X_i)}\right ],\] and each term of the second sum is \(\int _A f''_\theta (x)\,dx = \frac {\partial ^2}{\partial \theta ^2}\int _A f_\theta (x)\,dx = 0\). So \(J(\theta ) = E(S^2)\).

Solution. From example 2.1.3, \(S(\theta ) = -n + \frac 1\theta \sum X_i\) and \(J(\theta ) = \frac n\theta \). Then \[E(S) = -n + \frac {n\theta }{\theta } = 0,\qquad \var (S) = \frac {1}{\theta ^2}\sum \var (X_i) = \frac {n\theta }{\theta ^2} = \frac n\theta = J(\theta ).\]

Solution. \(\mathcal {L}(\theta ) = -n\ln \theta - \frac 1\theta \sum x_i\), so \[S(\theta ) = -\frac n\theta + \frac {\sum x_i}{\theta ^2},\qquad I(\theta ) = -\frac {n}{\theta ^2} + \frac {2\sum x_i}{\theta ^3} .\] The score equation gives \(\hat \theta = \bar x\), and \(I(\hat \theta ) = -\frac {n}{\bar x^2} + \frac {2n}{\bar x^2} = \frac {n}{\bar x^2}>0\). Since \(E(X_i) = \theta \), \[J(\theta ) = -\frac {n}{\theta ^2} + \frac {2n\theta }{\theta ^3} = \frac {n}{\theta ^2},\qquad \var (S) = \frac {1}{\theta ^4}\sum \var (X_i) = \frac {n\theta ^2}{\theta ^4} = \frac {n}{\theta ^2} .\] Note that \(I(\theta )\) depends on the data and can be negative for \(\theta \) far from \(\hat \theta \); only its expectation \(J(\theta )\) is guaranteed to be positive.

Solution. Write \(\theta _1 = \mu \) and \(\theta _2 = \sigma ^2\). Then \[\mathcal {L}(\theta ) = -\frac n2\ln (2\pi \theta _2) - \frac {1}{2\theta _2}\sum ^n_{i = 1}(x_i - \theta _1)^2,\] and the score is \[S(\theta ) = \begin {pmatrix} \frac {1}{\theta _2}\sum (x_i - \theta _1)\\[1ex] -\frac {n}{2\theta _2} + \frac {1}{2\theta _2^2}\sum (x_i - \theta _1)^2 \end {pmatrix}.\] Setting the first component to zero gives \(\hat \theta _1 = \bar x\); substituting into the second gives \(\hat \theta _2 = \frac 1n\sum (x_i - \bar x)^2\). So \[\hat \theta = \left (\bar X, \frac 1n\sum (X_i - \bar X)^2\right )^{\mathsf T} = \left (\bar X, \frac {n - 1}{n}S^2\right )^{\mathsf T} .\] Differentiating again, \[I(\theta ) = \begin {pmatrix} \frac {n}{\theta _2} & \frac {1}{\theta _2^2}\sum (x_i - \theta _1)\\[1ex] \frac {1}{\theta _2^2}\sum (x_i - \theta _1) & -\frac {n}{2\theta _2^2} + \frac {1}{\theta _2^3}\sum (x_i - \theta _1)^2 \end {pmatrix}.\] Since \(E(X_i - \theta _1) = 0\) and \(E(X_i - \theta _1)^2 = \theta _2\), \[J(\theta ) = \begin {pmatrix} \frac {n}{\theta _2} & 0\\[1ex] 0 & -\frac {n}{2\theta _2^2} + \frac {n}{\theta _2^2} \end {pmatrix} = \begin {pmatrix} \frac {n}{\sigma ^2} & 0\\[1ex] 0 & \frac {n}{2\sigma ^4} \end {pmatrix}.\] At \(\hat \theta \) the off-diagonal entries of \(I\) vanish and the diagonal entries are \(\frac {n}{\hat \sigma ^2}\) and \(\frac {n}{2\hat \sigma ^4}\), so \(I(\hat \theta )\) is positive definite and \(\hat \theta \) is a maximum. The ML estimator of \(\sigma ^2\) is biased, with mean \(\frac {n - 1}{n}\sigma ^2\).

Solution. \[\mathcal {L}(\theta ) = n\ln \theta + (\theta - 1)\sum \ln x_i - \sum x_i^\theta ,\] \[S(\theta ) = \frac n\theta + \sum \ln x_i - \sum x_i^\theta \ln x_i,\qquad I(\theta ) = \frac {n}{\theta ^2} + \sum x_i^\theta (\ln x_i)^2 .\] The equation \(S(\theta ) = 0\) cannot be solved explicitly, but \(I(\theta )>0\) for every \(\theta \), so \(\mathcal {L}\) is strictly concave and has exactly one maximum, which (2.2) finds from any reasonable start.

For the data, \(n = 5\) and \(\sum \ln x_i = -0.7624\). Starting at \(\theta ^{(0)} = 1\):

So \(\hat \theta = 1.995\), and \(\frac {1}{\sqrt {I(\hat \theta )}} = 0.613\) is its approximate standard error (section 2.5).

Solution. \[\mathcal {L}(\theta ) = (\alpha - 1)\sum \ln x_i - \frac 1\beta \sum x_i - n\alpha \ln \beta - n\ln \Gamma (\alpha ).\] With \(\psi (\alpha ) = \frac {d}{d\alpha }\ln \Gamma (\alpha )\), the digamma function, the score is \[S(\theta ) = \begin {pmatrix} \sum \ln x_i - n\ln \beta - n\psi (\alpha )\\[1ex] \frac {1}{\beta ^2}\sum x_i - \frac {n\alpha }{\beta } \end {pmatrix}.\] The second equation gives \(\beta = \frac {\bar x}{\alpha }\); substituting into the first leaves the single equation \[\ln \alpha - \psi (\alpha ) = \ln \bar x - \frac 1n\sum \ln x_i ,\] which has no closed-form solution and is solved by Newton’s method. (The right side is the log of the ratio of the arithmetic to the geometric mean, which is positive unless all the \(x_i\) are equal.) The information matrix is \[I(\theta ) = \begin {pmatrix} n\psi '(\alpha ) & \frac n\beta \\[1ex] \frac n\beta & \frac {2}{\beta ^3}\sum x_i - \frac {n\alpha }{\beta ^2} \end {pmatrix},\] and since \(E\left (\sum X_i\right ) = n\alpha \beta \), \[J(\theta ) = n\begin {pmatrix} \psi '(\alpha ) & \frac 1\beta \\[1ex] \frac 1\beta & \frac {\alpha }{\beta ^2} \end {pmatrix}.\] Here \(\psi '\) is the trigamma function. The iteration is \(\theta ^{(i+1)} = \theta ^{(i)} + \left [I(\theta ^{(i)})\right ]^{-1}S(\theta ^{(i)})\), started for example from the method of moments estimates \(\alpha ^{(0)} = \frac {\bar x^2}{s^2}\) and \(\beta ^{(0)} = \frac {s^2}{\bar x}\).