12.5 Conditionally linear parameters

Many non-linear models are linear in some of their parameters. In \(\theta _1e^{-\theta _2x}\) and \(\theta _1x/(\theta _2 + x)\), once \(\theta _2\) is fixed the model is a regression through the origin on a single known regressor. This can be used to reduce the search to the non-linear parameters alone.

Theorem 12.5.1 (Profile least squares). Suppose \(\mathbf {f}(\boldsymbol \theta ) = \mathbf {G}(\boldsymbol \phi )\boldsymbol \beta \), where \(\boldsymbol \theta = (\boldsymbol \beta , \boldsymbol \phi )\) and \(\mathbf {G}(\boldsymbol \phi )\) has full column rank for each \(\boldsymbol \phi \). For fixed \(\boldsymbol \phi \), \(S\) is minimised over \(\boldsymbol \beta \) by \(\hat {\boldsymbol \beta }(\boldsymbol \phi ) = (\mathbf {G}^\top \mathbf {G})^{-1}\mathbf {G}^\top \mathbf {y}\), with minimum \[ S_p(\boldsymbol \phi ) = \mathbf {y}^\top \bigl [\mathbf {I} - \mathbf {G}(\mathbf {G}^\top \mathbf {G})^{-1}\mathbf {G}^\top \bigr ]\mathbf {y} . \] The least-squares estimate is \(\hat {\boldsymbol \phi }\), the minimiser of \(S_p\), together with \(\hat {\boldsymbol \beta }(\hat {\boldsymbol \phi })\). With a single linear parameter, \(S_p(\phi ) = \mathbf {y}^\top \mathbf {y} - (\mathbf {g}^\top \mathbf {y})^2/\mathbf {g}^\top \mathbf {g}\).

Proof. For fixed \(\boldsymbol \phi \) the model is linear with model matrix \(\mathbf {G}(\boldsymbol \phi )\), so the minimum over \(\boldsymbol \beta \) is the least-squares fit and the minimum value is the residual sum of squares \(\mathbf {y}^\top (\mathbf {I} - \mathbf {H})\mathbf {y}\) (Theorem 3.4.2). Then \(\min _{\boldsymbol \beta ,\boldsymbol \phi }S = \min _{\boldsymbol \phi }\min _{\boldsymbol \beta }S = \min _{\boldsymbol \phi }S_p(\boldsymbol \phi )\). With one column \(\mathbf {g}\), \(\mathbf {y}^\top \mathbf {Hy} = (\mathbf {g}^\top \mathbf {y})^2/\mathbf {g}^\top \mathbf {g}\). □

With one non-linear parameter, \(S_p\) is a function of one variable that can simply be plotted, which shows at once whether there is a single minimum and roughly where it is (Figure 27). This is also the way to fit a linear spline with an unknown knot (Section 9): for each trial knot \(t\) the model is linear, and \(t\) is chosen to minimise the residual sum of squares (Problem 12.6.3).

𝜃S000001234𝜃ss2p.....0000ˆ2tt(12345 =arep𝜃ts20 1).3,0 24,7 ...

Figure 27: The profile residual sum of squares \(S_p(\theta _2)\) for the drug data, with \(\theta _1\) eliminated by Theorem 12.5.1. It has a single, sharp minimum at \(\hat \theta _2 = 0.3047\). The dots below the axis mark the values of \(\theta _2\) reached by the Gauss–Newton iterations of Example 12.3.2.

Example 12.5.2. The rate \(y\) of an enzyme reaction was measured at six concentrations \(x\) of its substrate:

\(x\) 0.5 1 2 4 8 16
\(y\) 2.3 3.1 5.2 6.4 8.2 8.8

Fit the Michaelis–Menten model \(y = \theta _1x/(\theta _2 + x) + \varepsilon \), first by the Lineweaver–Burk regression and then by least squares, and compare.

Solution. Lineweaver–Burk. Regressing \(1/y\) on \(1/x\) (with \(1/x = 2, 1, 0.5, 0.25, 0.125, 0.0625\) and \(1/y = 0.4348, 0.3226, 0.1923, 0.1563, 0.1220, 0.1136\)) gives \(1/y = 0.1115 + 0.1708(1/x)\), so \[ \tilde \theta _1 = \frac {1}{0.1115} = 8.968, \qquad \tilde \theta _2 = \frac {0.1708}{0.1115} = 1.532 . \] On the original scale these leave \(S = 1.057\).

Least squares. By Theorem 12.5.1, with \(g_i = x_i/(\theta _2 + x_i)\), \(\hat \theta _1(\theta _2) = \sum g_iy_i/\sum g_i^2\). A few trial values give

\(\theta _2\) \(\hat \theta _1(\theta _2)\) \(S_p(\theta _2)\)
1.5 9.299 0.754
2.0 9.994 0.303
2.5 10.614 0.694

so the minimum is near \(\theta _2 = 2\). Gauss–Newton started from the Lineweaver–Burk values converges in four steps to \[ \hat \theta _1 = 9.938, \quad \hat \theta _2 = 1.958, \qquad S(\hat {\boldsymbol \theta }) = 0.300 , \] with standard errors \(0.328\) and \(0.208\) from \(s^2 = 0.300/4 = 0.075\). The fitted rates are \(2.02\), \(3.36\), \(5.02\), \(6.67\), \(7.98\), \(8.86\).

Comparison. The Lineweaver–Burk estimates leave a residual sum of squares three and a half times as large, and they lie two to three standard errors from the least-squares estimates. The reason is the transformation: the smallest rates, at the smallest concentrations, become the largest values of \(1/y\) and \(1/x\), and their errors are magnified most, so they dominate the linear fit. The transformation is useful for a starting value and for a plot, but not for the final estimates.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.