4.5 Score and Wald Tests
Theorem 4.5.1 (Score Test). Score tests can be viewed as a more general class of tests of \(H_0:\, \theta = \theta _0\)
against \(H_1:\, \theta \in \Omega - \{\theta _0\}\).
If the usual regularity conditions hold then under \(H_0\) we have
\[S(\theta _0, X)\, \left [J(\theta _0)\right ]^{-\frac {1}{2}}\, \underset {D}{\longrightarrow }\, Z\sim N(0,1)\]
and thus
\[R(\theta _0,X) = \left [S(\theta _0,X)\right ]^2\, \left [J(\theta _0)\right ]^{-1}\, \underset {D}{\longrightarrow }\, W\thicksim \chi ^2_{(1)}.\]
For a vector \(\theta = (\theta _1, \, \cdots \, , \theta _k)^t\) we have
\[R(\theta _0, X) = \left [S(\theta _0, X)\right ]^t\, \left [J(\theta _0)\right ]^{-1}\, \left [S(\theta _0,X)\right ]\,\underset {D}{\longrightarrow }\ W\sim \chi ^2_{(k)}.\]
The test based on \(R(\theta _0,X)\) is called a (Rao) Score test. It has critical region
\(R =\{x:\, R(\theta _0, x)\, > c\}\) where \(c\) is determined by the size of the test i.e \(\alpha = P(W>c)\) where \(W\sim \chi ^2_{(k)}\). The test based on \(R(\theta _0,X)\) is asymptotically
equivalent to the likelihood ratio test.
Proof. Take the scalar case; the vector case differs only in bookkeeping. For a random sample the score is a sum of independent, identically distributed contributions, \[S(\theta _0,X) = \sum _{i=1}^{n} S_1(\theta _0,X_i),\] and under the regularity conditions each term has, at the true parameter, \[E_{\theta _0}\left [S_1(\theta _0,X_i)\right ] = 0, \qquad \var _{\theta _0}\left [S_1(\theta _0,X_i)\right ] = I(\theta _0),\] these being exactly the two identities established in the section on properties of the score. Hence \(E_{\theta _0}\left [S\right ]=0\) and \(\var _{\theta _0}\left [S\right ] = n\,I(\theta _0) = J(\theta _0)\).
Under \(H_0\) the sample is drawn from \(f_{\theta _0}\), so the central limit theorem applies to that sum and \[\frac {S(\theta _0,X)}{\sqrt {J(\theta _0)}} = S(\theta _0,X)\left [J(\theta _0)\right ]^{-\frac 12} \underset {D}{\longrightarrow } Z \sim N(0,1).\] The map \(z\mapsto z^{2}\) is continuous, so by part (2) of the limit theorems \[R(\theta _0,X) = \left [S(\theta _0,X)\right ]^{2}\left [J(\theta _0)\right ]^{-1} \underset {D}{\longrightarrow } Z^{2} \sim \chi ^{2}_{(1)},\] since the square of a standard normal variable has the chi-square distribution on one degree of freedom.
For \(\theta \in \mathbb {R}^{k}\) the multivariate central limit theorem gives \(J(\theta _0)^{-1/2}S(\theta _0,X) \underset {D}{\longrightarrow } Z\sim N_k(0,I)\), and the quadratic form \(Z^{t}Z\) has the \(\chi ^{2}_{(k)}\) distribution, which is the stated result. □
Note. The score test is computed entirely at \(\theta _0\): it never requires the maximum likelihood estimate. That is its practical advantage over the Wald and likelihood ratio tests, both of which need \(\widehat {\theta }\) first. Where the likelihood is awkward to maximise but easy to differentiate, this is the test to reach for.
Theorem 4.5.2 (Wald Test). Suppose that \(\widehat {\theta }\) is the ML estimator of \(\theta \) over all \(\theta \in \Omega \) and we wish to test \(H_0:\, \theta = \theta _0\) against \(H_1:\, \theta \in \Omega - \{\theta _0\}\). If the usual regularity conditions hold then under \(H_0\) \[w(\theta _0,X) = \left (\widehat {\theta } - \theta _0\right )^t \, J(\widehat {\theta })\, \left (\widehat {\theta } - \theta _0\right ) \, \underset {D}{\longrightarrow }\, W\sim \chi ^2_{(k)}.\] A test based on the test statistic \(w\left (\theta _0, X\right )\) is called a Wald test. It has critical region \[R = \{x:\, w\left (\theta _0, x\right ) > c\}\] where \(c\) is determined by the size of test.
Note. \(J(\theta _0)\) may be replaced by \(I\left (\widehat {\theta }\right )\) to give an asymptotically equivalent test statistic.
Example 4.5.3. Suppose \(X\thicksim BIN(n,\theta )\). Find the Score test and the Wald test for testing \(H_0:\, \theta = \theta _0\) against \(H_1:\, \theta \neq \theta _0\).
Problem 4.5.1. Suppose \((X_1, \, \cdots \, , X_n)\) is a random sample from the \(POI(\theta )\) distribution. Find the Score test and Wald test for testing \(H_0:\, \theta = \theta _0\) against \(H_1:\, \theta \neq \theta _0\).
Show solution
Solution. With \(T=\sum _{i=1}^{n}X_i\), \[\log L(\theta ) = -n\theta + T\log \theta + \text {const},\qquad S(\theta ,X) = \frac {T}{\theta } - n = \frac {T-n\theta }{\theta },\] and \(J(\theta ) = n/\theta \). The maximum likelihood estimator is \(\widehat {\theta }=\overline {X}\).
Score test
\[R(\theta _0,X) = \frac {\left [S(\theta _0,X)\right ]^{2}}{J(\theta _0)} = \frac {(T-n\theta _0)^{2}}{\theta _0^{2}}\cdot \frac {\theta _0}{n} = \frac {(T-n\theta _0)^{2}}{n\theta _0} = \frac {n\left (\overline {X}-\theta _0\right )^{2}}{\theta _0},\] and \(H_0\) is rejected when \(R>\chi ^{2}_{1,\,1-\alpha }\).
Wald test
\[w(\theta _0,X) = \left (\widehat {\theta }-\theta _0\right )^{2}J\left (\widehat {\theta }\right ) = \left (\overline {X}-\theta _0\right )^{2}\frac {n}{\overline {X}} = \frac {n\left (\overline {X}-\theta _0\right )^{2}}{\overline {X}},\] rejected on the same scale.
Note. The two differ only in where the information is evaluated — at \(\theta _0\) for the score test, at \(\widehat {\theta }\) for the Wald test — and they agree to first order, as the theory says they must. The score version needs no maximum likelihood estimate at all, and its null distribution uses only \(\theta _0\), which is why it is preferred when the null value is the natural reference point.
Solution. \(H_0:\, \theta = \theta _0\,\) against \(\, H_1:\, \theta \neq \theta _0\) \[L(\theta ) = \binom {n}{x}\, \theta ^x\, (1 - \theta )^{n - x}.\]
\[\mathcal {L}(\theta ) = \log \binom {n}{x} + x\, \log \theta + (n - x)\, \log (1 - \theta ).\]
\[S(\theta ) = \frac {x}{\theta } + (n - x)\, \frac {(1)}{1 - \theta } \, = \, \frac {x}{\theta } - \frac {n - x}{1 - \theta }.\]
\(S(\theta ) = 0\) \[\frac {x}{\theta } = \frac {n - x}{1 - \theta }\] \[x - \theta x = n\theta - \theta x\] \[\widehat {\theta } = \frac {x}{n} = \overline {x}.\]
\[I(\theta ) = -\left [\frac {-x}{\theta ^2} - \frac {(n - x)\, (-1)\, (-1)}{(1 - \theta )^2}\right ]\, = \, \frac {x}{\theta ^2} + \frac {n - x}{(1 - \theta )^2}.\] \begin {align*} J(\theta ) & = E\left [I(\theta ,X)\right ] = E\left [\frac {x}{\theta ^2} + \frac {n - x}{(1 - \theta )^2}\right ]\\ & = \frac {E(X)}{\theta ^2} + \frac {n - E(X)}{(1 - \theta )^2}\\ & =\frac {n\theta }{\theta ^2} + \frac {n - n\theta }{(1 - \theta )^2} \\ & = \frac {n}{\theta } + \frac {n}{1 -\theta }\\ & = \frac {n}{\theta (1 - \theta )}. \end {align*}
Score test: \begin {align*} R(\theta _0,x) & = \frac {\left (S(\theta _0,x)\right )^2}{J(\theta _0)} = \frac {\left (\frac {x}{\theta _0} - \frac {n - x}{1 - \theta _0}\right )^2}{\frac {n}{\theta _0(1 - \theta _0)}}\\ & = \left (\frac {x}{\theta _0} - \frac {n - x}{1 - \theta _0}\right )^2\, \frac {\theta _0\,(1 - \theta _0)}{n}\\ & = \left (\frac {x - n\theta _0}{\theta _0(1 - \theta _0)}\right )^2\, \frac {\theta _0(1 - \theta _0)}{n}\\ & = \frac {(x - n\theta _0)^2}{n\theta _0(1 - \theta _0)}. \end {align*}
with rejection region \(R = \{x:\, R(\theta _0,x)>c\}\) where \(c\) is determined from \(\alpha = P\left (\chi ^2_{(1)} > c\right )\).
Wald test: \begin {align*} w\left (\theta _0,x\right ) & = \left (\widehat {\theta } -\theta _0\right )^t\, J(\theta _0)\, \left (\widehat {\theta } - \theta _0\right )\\ & = \left (\frac {x}{n} - \theta _0\right )\, \frac {n}{\theta _0(1 - \theta _0)}\, \left (\frac {x}{n} - \theta _0\right )\\ & = \left (\frac {x}{n} - \theta _0\right )^2\, \frac {n}{\theta _0\, (1 - \theta _0)}\\ & = \left (\frac {x - n\theta _0}{n}\right )^2\, \frac {n}{\theta _0\, (1 -\theta _0)}\\ & = \frac {\left (x - n\theta _0\right )^2}{n\theta _0(1 - \theta _0)} \end {align*}
Therefore \(\, R = \{x:\, w(\theta _0, x) > c \}\,\) where \(c\) is determined from \(\alpha = P\left (\chi ^2_{(1)} > c\right )\,\)
i.e \(w(\theta _0, x) = R(\theta _0, x)\). □
Problem 4.5.2. Let \(X_1, \, \cdots \, , X_n\) be a random sample from the distribution with pdf \[f_{\theta }(x) = \frac {\theta }{x^{1 + \theta }}\, , \hspace {0.3cm} x\geq 1\] Find the Score test and the Wald test for testing \(H_0:\, \theta = \theta _0\) against \(H_1:\, \theta \neq \theta _0.\)
Show solution
Solution. For \(f_{\theta }(x)=\theta x^{-(1+\theta )}\) on \(x\geq 1\), write \(T=\sum _{i=1}^{n}\log X_i\). Then \[\log L(\theta ) = n\log \theta - (1+\theta )T,\qquad S(\theta ,X) = \frac {n}{\theta } - T,\] \[-\frac {\partial ^{2}\log L}{\partial \theta ^{2}} = \frac {n}{\theta ^{2}} \quad \Longrightarrow \quad J(\theta ) = \frac {n}{\theta ^{2}},\] the information being free of the data here. Setting the score to zero gives \(\widehat {\theta } = n/T\).
Score test
\[R(\theta _0,X) = \frac {\left [S(\theta _0,X)\right ]^{2}}{J(\theta _0)} = \left (\frac {n}{\theta _0}-T\right )^{2}\frac {\theta _0^{2}}{n} = \frac {\left (n-\theta _0 T\right )^{2}}{n} .\]
Wald test
\[w(\theta _0,X) = \left (\widehat {\theta }-\theta _0\right )^{2}J\left (\widehat {\theta }\right ) = \left (\frac {n}{T}-\theta _0\right )^{2}\frac {n}{\left (n/T\right )^{2}} = \frac {\left (n-\theta _0T\right )^{2}}{n} .\]
Both are referred to \(\chi ^{2}_{(1)}\) and \(H_0\) is rejected for large values.
Remark. The two statistics are identical here, which does not happen in general. It occurs because the information \(n/\theta ^{2}\) and the score \(n/\theta - T\) combine so that the \(\theta \) at which the information is evaluated cancels exactly. Recalling that \(T\sim \) GAM\((n,1/\theta )\), the common statistic is \(n\left (1-\theta _0T/n\right )^{2}\), a squared standardised deviation of \(T\) from its null mean \(n/\theta _0\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.