4.4 Likelihood Ratio Tests

For composite hypotheses the Neyman–Pearson ratio is replaced by a ratio of maximised likelihoods.

Definition 4.4.1. The likelihood ratio test of \(H_0\colon \theta \in \Omega _0\) against \(H_1\colon \theta \in \Omega - \Omega _0\) has critical region \(R = \{x : \Lambda (x)>C\}\), where \[\Lambda (x) = \frac {\sup _{\theta \in \Omega }L(\theta )}{\sup _{\theta \in \Omega _0}L(\theta )} = \frac {L(\hat \theta )}{L(\hat \theta _R)}\] and \(C\) is determined by the size. Here \(\hat \theta \) is the usual ML estimate and \(\hat \theta _R\) is the ML estimate under the restriction \(\theta \in \Omega _0\).

Example 4.4.2. Let \(X_1, \dots , X_n\) be a random sample from \(N(\mu , \sigma ^2)\) with both parameters unknown, and test \(H_0\colon \mu = 0\) against \(H_1\colon \mu \ne 0\). Show that the likelihood ratio test has critical region \(\left \{x : \frac {n\bar x^2}{s^2}>C\right \}\), that \(\frac {n\bar X^2}{S^2}\sim F(1, n - 1)\) under \(H_0\), and find the test of size \(0.05\) for \(n = 20\).

Solution. Here \(\Omega = \{(\mu , \sigma ^2) : \mu \in \mathbb {R}, \sigma ^2>0\}\) and \(\Omega _0 = \{(0, \sigma ^2) : \sigma ^2>0\}\). From example 2.2.4, the unrestricted ML estimates are \(\hat \mu = \bar x\) and \(\hat \sigma ^2 = \frac 1n\sum (x_i - \bar x)^2\), and substituting them, \[L(\hat \theta ) = (2\pi \hat \sigma ^2)^{-n/2}\exp \left \{-\frac {n\hat \sigma ^2}{2\hat \sigma ^2}\right \} = (2\pi \hat \sigma ^2)^{-n/2}e^{-n/2} .\] Under \(H_0\), \(\mathcal {L} = -\frac n2\ln (2\pi \sigma ^2) - \frac {1}{2\sigma ^2}\sum x_i^2\); setting \(\frac {\partial \mathcal {L}}{\partial \sigma ^2} = -\frac {n}{2\sigma ^2} + \frac {1}{2\sigma ^4}\sum x_i^2 = 0\) gives \(\hat \sigma _R^2 = \frac 1n\sum x_i^2\), and in the same way \(L(\hat \theta _R) = (2\pi \hat \sigma _R^2)^{-n/2}e^{-n/2}\). Since \(\sum x_i^2 = \sum (x_i - \bar x)^2 + n\bar x^2 = (n - 1)s^2 + n\bar x^2\), \[\Lambda (x) = \left (\frac {\hat \sigma _R^2}{\hat \sigma ^2}\right )^{n/2} = \left (\frac {(n - 1)s^2 + n\bar x^2}{(n - 1)s^2}\right )^{n/2} = \left (1 + \frac {1}{n - 1}\cdot \frac {n\bar x^2}{s^2}\right )^{n/2} .\] This increases with \(\frac {n\bar x^2}{s^2}\), so \(\Lambda >K\) is equivalent to \(\frac {n\bar x^2}{s^2}>C\).

Under \(H_0\), \[T = \frac {n\bar X^2}{S^2} = \frac {\left (\frac {\bar X}{\sigma /\sqrt n}\right )^2\big /1}{\frac {(n - 1)S^2}{\sigma ^2}\big /(n - 1)},\] a ratio of a \(\chi ^2(1)\) variable to an independent \(\chi ^2(n - 1)\) variable, each divided by its degrees of freedom, so \(T\sim F(1, n - 1)\). For \(n = 20\), \(0.05 = P(F(1, 19)>C)\) gives \(C = 4.38\). Since \(T\) is the square of the one-sample \(t\) statistic and \(t_{19}(0.025)^2 = 2.093^2 = 4.38\), this is the two-sided \(t\) test.

When the exact distribution of \(\Lambda \) is not available, the following large-sample result is used. It is proved in problem 4.4.3.

Theorem 4.4.3 (Wilks). Suppose the model is regular, \(\Omega \) is an open subset of \(\mathbb {R}^k\), and \(\Omega _0 = \{\theta (\eta ) : \eta \in E\}\) for an open set \(E\subset \mathbb {R}^q\) and a smooth one-to-one map \(\theta (\cdot )\), with \(q<k\). Then under \(H_0\), \[2\ln \Lambda (X)\underset {D}{\longrightarrow }\chi ^2(k - q) .\]

The number of degrees of freedom, \(k - q\), is the number of parameters that must be estimated in the general model minus the number that must be estimated under \(H_0\). The approximate test of size \(\alpha \) rejects \(H_0\) when \(2\ln \Lambda >\chi ^2_{k - q}(\alpha )\).

Example 4.4.4. Let \(X_1, \dots , X_n\) be independent \(\text {POI}(\lambda )\) and, independently, \(Y_1, \dots , Y_n\) be independent \(\text {POI}(\mu )\). Find the likelihood ratio test of \(H_0\colon \lambda = \mu \) against \(H_1\colon \lambda \ne \mu \), and apply it when \(n = 20\), \(\bar x = 3.1\) and \(\bar y = 2.2\).

Solution. Here \(\Omega = \{(\lambda , \mu ) : \lambda , \mu >0\}\), with \(k = 2\), and \(\Omega _0 = \{(\lambda , \lambda ) : \lambda >0\}\), with \(q = 1\). Dropping the factorials, which cancel in the ratio, \[L(\lambda , \mu )\propto e^{-n\lambda }\lambda ^{n\bar x}e^{-n\mu }\mu ^{n\bar y} .\] The unrestricted ML estimates are \(\hat \lambda = \bar x\) and \(\hat \mu = \bar y\). Under \(H_0\), \(L\propto e^{-2n\lambda }\lambda ^{n(\bar x + \bar y)}\), maximised at \(\hat \lambda _R = \frac {\bar x + \bar y}{2} = m\), say. The factors \(e^{-n(\bar x + \bar y)}\) appear in both maxima and cancel, leaving \[\Lambda = \frac {\bar x^{n\bar x}\bar y^{n\bar y}}{m^{n(\bar x + \bar y)}} = \left (\frac {\bar x^{\bar x}\bar y^{\bar y}}{m^{\bar x + \bar y}}\right )^n,\qquad 2\ln \Lambda = 2n\left [\bar x\ln \frac {\bar x}{m} + \bar y\ln \frac {\bar y}{m}\right ].\] The test rejects \(H_0\) when this is large. By theorem 4.4.3, under \(H_0\), \(2\ln \Lambda \) is approximately \(\chi ^2(2 - 1) = \chi ^2(1)\).

For the data, \(m = 2.65\) and \[2\ln \Lambda = 40\left [3.1\ln \frac {3.1}{2.65} + 2.2\ln \frac {2.2}{2.65}\right ] = 40\,[0.48621 - 0.40943] = 3.071 .\] Since \(3.071<\chi ^2_1(0.05) = 3.841\), \(H_0\) is not rejected at the \(5\%\) level; the approximate \(p\)-value is \(P(\chi ^2(1)>3.071) = 0.080\).

Problem 4.4.1. Suppose \(X\thicksim GAM(2,\beta _1)\) and \(Y\thicksim GAM(2,\beta _2)\) independently. Show that the likelihood ratio statistic for testing the hypothesis \(H_0: \beta _1 = \beta _2\,\) against the alternative \(\, H_1:\beta _1\neq \beta _2\) is a function of the statistic \[T = \frac {X}{X + Y}.\] Find the distribution of \(T\) under \(H_0\). Find the critical region for a size \(\alpha = 0.01\) test.

Hint: If \(X\thicksim GAM(a,1)\) and \(Y\thicksim GAM(b,1)\) are independent random variable then \[\frac {X}{X + Y} \thicksim BETA(a,b).\]

Show solution

Solution. With \(X\sim \) GAM\((2,\beta _1)\) and \(Y\sim \) GAM\((2,\beta _2)\) independent, \[L(\beta _1,\beta _2) = \frac {x\,e^{-x/\beta _1}}{\beta _1^{2}}\cdot \frac {y\,e^{-y/\beta _2}}{\beta _2^{2}} .\] Unrestricted, the maximum likelihood estimates are \(\widehat {\beta }_1 = x/2\) and \(\widehat {\beta }_2 = y/2\). Under \(H_0:\beta _1=\beta _2=\beta \) the total \(x+y\) has shape \(4\), so \(\widehat {\beta } = (x+y)/4\). Substituting, \[\Lambda = \frac {\sup _{H_0}L}{\sup L} = \frac {\left [(x+y)/4\right ]^{-4}e^{-4}} {\left (x/2\right )^{-2}\left (y/2\right )^{-2}e^{-2}e^{-2}} = \frac {\left (x/2\right )^{2}\left (y/2\right )^{2}}{\left [(x+y)/4\right ]^{4}} = \frac {64\,x^{2}y^{2}}{(x+y)^{4}} .\] Writing \(t = x/(x+y)\), so that \(x=t(x+y)\) and \(y=(1-t)(x+y)\), the factors of \((x+y)\) cancel and \[\Lambda = 64\,t^{2}(1-t)^{2} = 64\left [t(1-t)\right ]^{2},\] a function of \(T\) alone.

The distribution of \(T\) under \(H_0\)

Under \(H_0\), \(X\) and \(Y\) are independent gamma variables with the same scale and shapes \(2\) and \(2\), so \[T = \frac {X}{X+Y} \ \sim \ \text {BETA}(2,2),\] with density \(6t(1-t)\) on \((0,1)\). The common scale \(\beta \) cancels, which is what makes the null distribution free of the nuisance parameter and the test usable.

Note. \(\Lambda \) is largest when \(t=\tfrac 12\) and falls away symmetrically, so rejecting for small \(\Lambda \) means rejecting when \(T\) is far from \(\tfrac 12\) in either direction — exactly what a two-sided test of \(\beta _1=\beta _2\) should do.

Problem 4.4.2. Suppose \(X_1, \, \cdots \, , X_n\) are independent \(EXP(\lambda )\) random variables and independently \(Y_1, \, \cdots \, , \, Y_n\) are independent \(EXP(\mu )\) random variables.
Show that the likelihood ratio statistic for testing the hypothesis \(H_0: \lambda = \mu \) against the alternative \(H_1: \lambda \neq \mu \) is a function of \[T = \frac {\sum ^n_{i = 1} X_i}{\sum ^n_{i =1} X_i + \sum ^n_{i = 1} Y_i}.\]

Show solution

Solution. Write \(S=\sum _{i=1}^{n}X_i\) and \(U=\sum _{i=1}^{n}Y_i\). With the exponential parameterised by its mean, \[L(\lambda ,\mu ) = \lambda ^{-n}e^{-S/\lambda }\,\mu ^{-n}e^{-U/\mu },\] so unrestricted \(\widehat {\lambda }=S/n\), \(\widehat {\mu }=U/n\), while under \(H_0:\lambda =\mu \) the pooled estimate is \(\widehat {\lambda }_0=(S+U)/(2n)\). Hence \[\Lambda = \frac {\left [(S+U)/2n\right ]^{-2n}e^{-2n}} {\left (S/n\right )^{-n}\left (U/n\right )^{-n}e^{-n}e^{-n}} = \frac {\left (S/n\right )^{n}\left (U/n\right )^{n}}{\left [(S+U)/2n\right ]^{2n}} = 2^{2n}\,\frac {S^{n}U^{n}}{(S+U)^{2n}} .\] Putting \(T = S/(S+U)\) gives \[\Lambda = 2^{2n}\left [T(1-T)\right ]^{n},\] so the statistic is a function of \(T\) alone — the same shape as the previous problem with \(n\) in place of \(1\).

Under \(H_0\) the sums \(S\) and \(U\) are independent GAM\((n,\lambda )\) variables with a common scale, so \[T = \frac {S}{S+U}\ \sim \ \text {BETA}(n,n),\] free of \(\lambda \). The test rejects for small \(\Lambda \), that is when \(T\) departs from \(\tfrac 12\) in either direction, with the critical values taken from the BETA\((n,n)\) distribution.

Problem 4.4.3. Suppose \(X_1, \, \cdots \, , X_n\) is a random sample from a regular statistical model \(\{f_{\theta }(x); \theta \in \Omega \}\) with \(\Omega \) being an open set in a \(k\)-dimensional Euclidean space. Consider a subset of \(\Omega \) defined by \[\Omega _0 = \{\theta (\eta ):\, \eta \in \, \text { open subset of a}\, q-\text {dimensional Euclidean space}\}.\] Then the likelihood ratio statistic defined by \[\Lambda _n(x) = \frac {\sup _{\theta \in \Omega } \prod ^n_{i = 1} f_{\theta }(x_i)}{\sup _{\theta \in \Omega _0} \prod ^n_{i =1} f_{\theta }(x_i)} = \frac {\sup _{\theta \in \Omega }L(\theta )}{\sup _{\theta \in \Omega _0} L(\theta )}\] is such that, under \(H_0: \theta \in \Omega _0\), \[2\, \log \Lambda _n(x) \, \underset {D}{\longrightarrow } \, W\thicksim \chi ^2_{(k - q)}.\]

Show solution

Solution. This is Wilks’ theorem. A full proof belongs to asymptotic theory, but the argument is short enough to give in outline, and it explains where the degrees of freedom come from.

Let \(\widehat {\theta }\) maximise the likelihood over \(\Omega \) and \(\widehat {\theta }_0\) over \(\Omega _0\), and expand \(\log L\) about \(\widehat {\theta }\) to second order. The first-order term vanishes because \(\widehat {\theta }\) is a stationary point, leaving \[2\log \Lambda _n = 2\left [\log L\left (\widehat {\theta }\right ) - \log L\left (\widehat {\theta }_0\right )\right ] \approx \left (\widehat {\theta }-\widehat {\theta }_0\right )^{t} J\left (\theta _0\right ) \left (\widehat {\theta }-\widehat {\theta }_0\right ),\] since \(-\partial ^{2}\log L/\partial \theta \partial \theta ^{t}\) converges to \(J(\theta _0)\).

Under \(H_0\) the large-sample theory of the previous section gives \[J\left (\theta _0\right )^{1/2}\left (\widehat {\theta }-\theta _0\right ) \underset {D}{\longrightarrow } Z \sim N_k(0,I),\] so the quadratic form above is asymptotically \(Z^{t}PZ\), where \(P\) is the orthogonal projection onto the orthogonal complement of the tangent space of \(\Omega _0\) at \(\theta _0\). That tangent space has dimension \(q\), so \(P\) has rank \(k-q\). A quadratic form \(Z^{t}PZ\) in a standard normal vector, with \(P\) an orthogonal projection of rank \(r\), has the \(\chi ^{2}_{(r)}\) distribution. Hence \[2\log \Lambda _n \underset {D}{\longrightarrow } W \sim \chi ^{2}_{(k-q)} .\]

Note. The degrees of freedom are the number of restrictions — the dimension the null hypothesis removes — and not the dimension of either space on its own. That is the practical content of the theorem: to use it, count constraints. The regularity conditions matter too: \(\Omega _0\) must be a smooth surface in the interior of \(\Omega \), which is why the result fails when the null puts a parameter on the boundary, as in testing a variance component equal to zero.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.