12.4 Inference
The linear approximation that drives Gauss–Newton also gives approximate inference.
Theorem 12.4.1 (Approximate distribution). For large samples, or small error variance, approximately \[ \hat {\boldsymbol \theta } \sim N\bigl (\boldsymbol \theta , \sigma ^2(\mathbf {F}^\top \mathbf {F})^{-1}\bigr ) , \] with \(\mathbf {F}\) evaluated at the true \(\boldsymbol \theta \), and \(s^2 = S(\hat {\boldsymbol \theta })/(n - p)\) is approximately unbiased for \(\sigma ^2\).
Derivation. Near the true \(\boldsymbol \theta \), \(\mathbf {f}(\boldsymbol \vartheta ) \approx \mathbf {f}(\boldsymbol \theta ) + \mathbf {F}(\boldsymbol \vartheta - \boldsymbol \theta )\), so \(\mathbf {y} - \mathbf {f}(\boldsymbol \theta ) = \boldsymbol \varepsilon \) is approximately a linear model \(\mathbf {F}(\boldsymbol \vartheta - \boldsymbol \theta ) + \boldsymbol \varepsilon \) in the unknown \(\boldsymbol \vartheta - \boldsymbol \theta \) with true value \(\mathbf {0}\). Minimising \(S\) near \(\boldsymbol \theta \) is then, approximately, least squares for this linear model, so \[ \hat {\boldsymbol \theta } - \boldsymbol \theta \approx (\mathbf {F}^\top \mathbf {F})^{-1}\mathbf {F}^\top \boldsymbol \varepsilon . \] This has mean \(\mathbf {0}\) and variance \(\sigma ^2(\mathbf {F}^\top \mathbf {F})^{-1}\) (Theorem 3.5.1), and is normal if \(\boldsymbol \varepsilon \) is. The residual vector is approximately \((\mathbf {I} - \mathbf {H}_F)\boldsymbol \varepsilon \) with \(\mathbf {H}_F = \mathbf {F}(\mathbf {F}^\top \mathbf {F})^{-1}\mathbf {F}^\top \), whose expected squared length is \((n - p)\sigma ^2\), as in Section 3. □
So, in practice, \(\widehat {\operatorname {Var}}(\hat {\boldsymbol \theta }) = s^2(\hat {\mathbf {F}}^\top \hat {\mathbf {F}})^{-1}\) with \(\hat {\mathbf {F}} = \mathbf {F}(\hat {\boldsymbol \theta })\), and the \(t\)-tests and \(t\)-intervals of Section 3 are used with \(n - p\) degrees of freedom. The quality of the approximation depends on how nearly linear the model is over the region where \(\hat {\boldsymbol \theta }\) is likely to fall, which in turn depends on the parametrisation: the same curve written with different parameters can give much better or much worse approximations. Approximate \(F\)-tests comparing nested models use the extra sum of squares, as before. Functions of the parameters, such as a half-life or a predicted value, are handled by the delta method.
Example 12.4.2. For the drug data of Example 12.3.2, find standard errors for \(\hat \theta _1\) and \(\hat \theta _2\), a \(95\%\) confidence interval for the elimination rate \(\theta _2\), and an interval for the half-life \(\log 2/\theta _2\).
Solution. \(s^2 = 0.1539/(6 - 2) = 0.0385\). Evaluating \(\mathbf {F}\) at \(\hat {\boldsymbol \theta }\) and inverting \(\hat {\mathbf {F}}^\top \hat {\mathbf {F}}\) gives \[ \operatorname {se}(\hat \theta _1) = 0.342 , \qquad \operatorname {se}(\hat \theta _2) = 0.00704 , \] with correlation \(0.85\) between the estimates: a larger initial concentration goes with a faster decay. With \(t_{0.025,4} = 2.776\), \[ 0.3047 - 2.776(0.00704) \le \theta _2 \le 0.3047 + 2.776(0.00704) , \] that is, \(0.285 \le \theta _2 \le 0.324\). The half-life is estimated as \(\log 2/0.3047 = 2.275\) hours. Its derivative with respect to \(\theta _2\) is \(-\log 2/\theta _2^2\), so by the delta method (Lemma 6.1.1) \[ \operatorname {se} = \frac {\log 2}{0.3047^2}(0.00704) = 0.0526 , \] and an approximate \(95\%\) interval is \(2.275 \pm 2.776(0.0526)\), from \(2.13\) to \(2.42\) hours. Since the half-life is a decreasing function of \(\theta _2\), transforming the end points of the interval for \(\theta _2\) gives a very similar answer, \((2.14, 2.43)\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.