Proof. Fix \(\varepsilon >0\) small enough that \([\theta _0-\varepsilon ,\theta _0+\varepsilon ]\subset \Omega \), and write \[\ell _n(\theta ) = \frac {1}{n}\sum _{i=1}^{n}\log f_{\theta }(X_i).\] For each fixed \(\theta \) the summands are independent and identically distributed, so by the weak law of large numbers \[\ell _n(\theta ) - \ell _n(\theta _0) \underset {P}{\longrightarrow } E_{\theta _0}\left [\log \frac {f_{\theta }(X)}{f_{\theta _0}(X)}\right ] = -K(\theta _0,\theta ),\] where \(K(\theta _0,\theta )\) is the Kullback–Leibler divergence. By Jensen’s inequality applied to the strictly concave logarithm, \[E_{\theta _0}\left [\log \frac {f_{\theta }(X)}{f_{\theta _0}(X)}\right ] \;\leq \; \log E_{\theta _0}\left [\frac {f_{\theta }(X)}{f_{\theta _0}(X)}\right ] = \log \int f_{\theta }(x)\,dx = 0,\] with equality only when \(f_{\theta } = f_{\theta _0}\) almost everywhere. Since the model is identifiable, \(K(\theta _0,\theta ) > 0\) for every \(\theta \neq \theta _0\).

Apply this at the two endpoints \(\theta _0 \pm \varepsilon \). With probability tending to one, \[\ell _n(\theta _0) > \ell _n(\theta _0-\varepsilon ) \qquad \text {and}\qquad \ell _n(\theta _0) > \ell _n(\theta _0+\varepsilon ),\] so \(\ell _n\) attains a local maximum at some interior point \(\widehat {\theta _n}\in (\theta _0-\varepsilon ,\theta _0+\varepsilon )\). At an interior maximum of a differentiable function the derivative vanishes, so \(\widehat {\theta _n}\) solves the likelihood equation \(\sum _{i=1}^{n}S_1(\theta ,X_i)=0\). As \(\varepsilon >0\) was arbitrary, \[P\left (\left |\widehat {\theta _n}-\theta _0\right |<\varepsilon \right )\longrightarrow 1 \qquad \text {for every } \varepsilon >0,\] which is exactly \(\widehat {\theta _n}\underset {P}{\longrightarrow }\theta _0\). □

Sketch of proof. Expand the score about \(\theta _0\). Since \(S(\hat \theta _n) = 0\) and \(S' = -I\), Taylor’s theorem gives \[0 = S(\hat \theta _n) = S(\theta _0) - (\hat \theta _n - \theta _0)I(\theta _n^*)\] for some \(\theta _n^*\) between \(\theta _0\) and \(\hat \theta _n\). Rearranging, \[\sqrt n\left (\hat \theta _n - \theta _0\right ) = \frac {S(\theta _0)/\sqrt n}{I(\theta _n^*)/n} .\] The numerator is \(\frac {1}{\sqrt n}\sum S_1(\theta _0; X_i)\), a normalised sum of independent terms with mean \(0\) and variance \(J_1(\theta _0)\) (theorem 2.2.1); by the central limit theorem it converges in distribution to \(N(0, J_1(\theta _0))\). In the denominator, \(\frac 1n I(\theta _0) = \frac 1n\sum I_1(\theta _0; X_i)\underset {P}{\longrightarrow }J_1(\theta _0)\) by the weak law of large numbers, and the bound on the third derivative together with \(\theta _n^*\underset {P}{\longrightarrow }\theta _0\) shows that replacing \(\theta _0\) by \(\theta _n^*\) changes the limit by a term tending to zero. By Slutsky’s theorem the ratio converges to \(\frac {1}{J_1(\theta _0)}N(0, J_1(\theta _0)) = N\left (0, \frac {1}{J_1(\theta _0)}\right )\). The statement for \(\tau (\hat \theta _n)\) then follows from the delta method, part (5) of the limit theorems with \(b = \frac 12\). □

Solution.

(a)

\(\mathcal {L}(\theta ) = n\ln \theta + (\theta - 1)\sum \ln x_i\), so \(S(\theta ) = \frac n\theta + \sum \ln x_i\) and \(\hat \theta _n = \frac nT\) with \(T = -\sum \ln X_i\).

(b)

\(I(\theta ) = \frac {n}{\theta ^2} = J(\theta )\), so the approximate interval is \[\hat \theta _n\pm 1.96\frac {\hat \theta _n}{\sqrt n} = \frac nT\pm 1.96\frac {\sqrt n}{T} .\]

(c)

As in example 2.3.4, \(T\sim \text {GAM}\left (n, \frac 1\theta \right )\), so \(Y = 2\theta T\sim \text {GAM}(n, 2) = \chi ^2(2n)\) is an exact pivot. Let \(a\) and \(b\) be the lower and upper \(2.5\%\) points of \(\chi ^2(2n)\), so that \(P(a<Y<b) = 0.95\), as in figure 2.5.2. Then \[0.95 = P(a<2\theta T<b) = P\left (\frac {a}{2T}<\theta <\frac {b}{2T}\right ),\] and \(\left [\frac {a}{2T}, \frac {b}{2T}\right ]\) is an exact \(95\%\) confidence interval.

(d)

\(\hat \theta = \frac {10}{4} = 2.5\). The approximate interval is \(2.5\pm 1.96\cdot \frac {2.5}{\sqrt {10}} = 2.5\pm 1.550\), that is \((0.95, 4.05)\). For \(\chi ^2(20)\), \(a = 9.591\) and \(b = 34.170\), so the exact interval is \(\left (\frac {9.591}{8}, \frac {34.170}{8}\right ) = (1.20, 4.27)\). The exact interval is not symmetric about \(\hat \theta \); it reflects the skewness of the sampling distribution, which the normal approximation ignores at this sample size.

Solution. By example 2.1.3, \(\hat \theta _n = \bar X\) and \(J(\theta ) = \frac n\theta \). The approximate interval for \(\theta \) is \[\bar x\pm 1.96\sqrt {\frac {\bar x}{n}} .\] For \(\tau (\theta ) = e^{-\theta }\), \(\tau '(\theta ) = -e^{-\theta }\), and by theorem 2.5.3 \(\tau (\hat \theta _n)\) is approximately normal with mean \(\tau (\theta )\) and variance \(\frac {[\tau '(\theta )]^2}{J(\theta )} = \frac {e^{-2\theta }\theta }{n}\). The interval is \[e^{-\bar x}\pm 1.96\,e^{-\bar x}\sqrt {\frac {\bar x}{n}} .\] For \(n = 50\) and \(\bar x = 2.4\), \(\sqrt {\bar x/n} = 0.2191\), so the interval for \(\theta \) is \(2.4\pm 0.429 = (1.97, 2.83)\), and the interval for \(e^{-\theta }\) is \(0.0907\pm 0.0390 = (0.052, 0.130)\). A slightly better interval for \(e^{-\theta }\) is obtained by transforming the endpoints of the first interval: \((e^{-2.83}, e^{-1.97}) = (0.059, 0.139)\), which, unlike the delta-method interval, can never contain negative values.

Solution. \[L(\theta ) = \prod ^n_{i = 1}e^{-(x_i - \theta )}I(x_i>\theta ) = e^{n\theta - \sum x_i}I\left (x_{(1)}>\theta \right ).\] This increases in \(\theta \) up to \(x_{(1)}\) and is zero beyond it (figure 2.2), so \(\hat \theta _n = X_{(1)}\).

Each \(X_i\) has \(P(X_i>y) = e^{-(y - \theta )}\) for \(y>\theta \), so \(P(X_{(1)}>y) = e^{-n(y - \theta )}\). Since \(X_{(1)}>\theta \) always, for \(\varepsilon >0\) \[P\left (\left |X_{(1)} - \theta \right |>\varepsilon \right ) = P\left (X_{(1)}>\theta + \varepsilon \right ) = e^{-n\varepsilon }\longrightarrow 0,\] and \(\hat \theta _n\) is consistent. Moreover, for \(t>0\), \[P\left (n(X_{(1)} - \theta )\le t\right ) = 1 - P\left (X_{(1)}>\theta + \frac tn\right ) = 1 - e^{-t},\] so \(Q = n(X_{(1)} - \theta )\sim \text {EXP}(1)\) exactly, whatever \(n\). It is an exact pivot, and the error of \(\hat \theta _n\) is of order \(\frac 1n\) rather than \(\frac {1}{\sqrt n}\): the model is not regular, and theorem 2.5.3 does not apply.

For an equal-tailed interval take \(a = -\ln 0.975 = 0.0253\) and \(b = -\ln 0.025 = 3.689\), so that \(P(a<Q<b) = 0.95\) (figure 2.3). Then \[0.95 = P\left (a<n(X_{(1)} - \theta )<b\right ) = P\left (X_{(1)} - \frac bn<\theta <X_{(1)} - \frac an\right ),\] giving the interval \(\left [X_{(1)} - \frac {3.689}{n},\ X_{(1)} - \frac {0.0253}{n}\right ]\). Because the density of \(Q\) is decreasing, the shortest \(95\%\) interval puts all the excluded probability in the upper tail: \(P(0<Q<-\ln 0.05) = 0.95\) gives \(\left [X_{(1)} - \frac {2.996}{n},\ X_{(1)}\right ]\), of length \(\frac {2.996}{n}\) against \(\frac {3.663}{n}\).

Solution. From example 2.2.6, \(J(\theta ) = n\begin {pmatrix}\psi '(\alpha ) & \frac 1\beta \\ \frac 1\beta & \frac {\alpha }{\beta ^2}\end {pmatrix}\), with determinant \(\frac {n^2\left (\alpha \psi '(\alpha ) - 1\right )}{\beta ^2}\), so \[J(\theta )^{-1} = \frac {\beta ^2}{n\left (\alpha \psi '(\alpha ) - 1\right )} \begin {pmatrix}\frac {\alpha }{\beta ^2} & -\frac 1\beta \\[1ex] -\frac 1\beta & \psi '(\alpha )\end {pmatrix}.\] The \((2, 2)\) entry is \(\frac {\beta ^2\psi '(\alpha )}{n\left (\alpha \psi '(\alpha ) - 1\right )}\), and the approximate interval for \(\beta \) is \[\hat \beta \pm 1.96\,\hat \beta \sqrt {\frac {\psi '(\hat \alpha )}{n\left (\hat \alpha \psi '(\hat \alpha ) - 1\right )}} .\] For \(\tau = \alpha \beta \), \(D(\theta ) = (\beta , \alpha )^{\mathsf T}\) and \[D^{\mathsf T}J^{-1}D = \frac {\beta ^2}{n\left (\alpha \psi '(\alpha ) - 1\right )} \left [\alpha - 2\alpha + \alpha ^2\psi '(\alpha )\right ] = \frac {\alpha \beta ^2}{n} .\] So the interval for the mean is \(\hat \alpha \hat \beta \pm 1.96\,\hat \beta \sqrt {\frac {\hat \alpha }{n}}\). This has a simple explanation: the second score equation of example 2.2.6 says \(\hat \alpha \hat \beta = \bar x\), and \(\var (\bar X) = \frac {\alpha \beta ^2}{n}\) exactly.