13.3 Estimating the parameters
If the autocorrelation comes from a missing regressor, the remedy is to find it and include it, and then to check the residuals again. If it is genuine AR(1) behaviour, the model can be transformed into one with independent errors.
Theorem 13.3.1 (Cochrane–Orcutt transformation). Under the AR(1) model, for \(t = 2, \dots , n\), let \(y_t' = y_t - \rho y_{t-1}\) and \(x_t' = x_t - \rho x_{t-1}\). Then \[ y_t' = \beta _0' + \beta _1x_t' + a_t , \qquad \beta _0' = \beta _0(1 - \rho ) , \] a regression with the same slope and with independent errors of constant variance \(\sigma _a^2\).
Proof. Subtract \(\rho \) times the model at time \(t - 1\) from the model at time \(t\): \[ y_t - \rho y_{t-1} = \beta _0(1 - \rho ) + \beta _1(x_t - \rho x_{t-1}) + (\varepsilon _t - \rho \varepsilon _{t-1}) , \] and \(\varepsilon _t - \rho \varepsilon _{t-1} = a_t\). □
So if \(\rho \) were known, ordinary least squares on the transformed data would satisfy all the standard assumptions, and the usual standard errors and tests would be valid. The transformation loses the first observation. The Prais–Winsten version keeps it, as \(y_1' = \sqrt {1 - \rho ^2}\,y_1\) with the same multiplier on its row of \(\mathbf {X}\).
Theorem 13.3.2. Let \(\mathbf {P}\) be the \(n \times n\) matrix with \(P_{11} = \sqrt {1 - \rho ^2}\), \(P_{tt} = 1\) and \(P_{t,t-1} = -\rho \) for \(t \ge 2\), and zeros elsewhere. Then \(\operatorname {Var}(\mathbf {P}\boldsymbol \varepsilon ) = \sigma _a^2\mathbf {I}\), so least squares on \(\mathbf {Py}\) and \(\mathbf {PX}\) is the generalized least-squares estimator \[ \hat {\boldsymbol \beta }_{\mathrm {GLS}} = (\mathbf {X}^\top \boldsymbol \Sigma ^{-1}\mathbf {X})^{-1}\mathbf {X}^\top \boldsymbol \Sigma ^{-1}\mathbf {y} , \] which has the smallest variance among linear unbiased estimators.
Proof. For \(t \ge 2\), the \(t\)th element of \(\mathbf {P}\boldsymbol \varepsilon \) is \(\varepsilon _t - \rho \varepsilon _{t-1} = a_t\). The first is \(\sqrt {1 - \rho ^2}\,\varepsilon _1\), with variance \((1 - \rho ^2)\sigma _a^2/(1 - \rho ^2) = \sigma _a^2\) by Theorem 13.1.1. The \(a_t\) for \(t \ge 2\) are independent of each other and of \(\varepsilon _1\), so \(\operatorname {Var}(\mathbf {P}\boldsymbol \varepsilon ) = \sigma _a^2\mathbf {I}\). Then \(\mathbf {P}\boldsymbol \Sigma \mathbf {P}^\top \sigma ^2 = \sigma _a^2\mathbf {I}\), so \(\boldsymbol \Sigma ^{-1} = (\sigma ^2/\sigma _a^2)\mathbf {P}^\top \mathbf {P}\), and least squares on the transformed data gives \((\mathbf {X}^\top \mathbf {P}^\top \mathbf {PX})^{-1}\mathbf {X}^\top \mathbf {P}^\top \mathbf {Py}\), the estimator displayed, since the constant cancels. This is weighted least squares with \(\mathbf {W} = \boldsymbol \Sigma ^{-1}\), so the optimality is Theorem 6.4.1(iii). □
In practice \(\rho \) is unknown and must be estimated. Three methods are common.
- Cochrane–Orcutt. Estimate \(\rho \) from the least-squares residuals by regressing \(e_t\) on \(e_{t-1}\) through the origin, \(\hat \rho = \sum _{t=2}^ne_te_{t-1}/\sum _{t=2}^ne_{t-1}^2\); transform; fit; and, if desired, recompute the residuals from the new estimates and repeat until \(\hat \rho \) settles.
- Hildreth–Lu. For each \(\rho \) on a grid, transform and fit, and choose the \(\rho \) with the smallest residual sum of squares. This is profile least squares (Theorem 12.5.1) with \(\rho \) as the non-linear parameter.
- First differences. When \(\hat \rho \) is close to \(1\), take \(\rho = 1\): regress \(y_t - y_{t-1}\) on \(x_t - x_{t-1}\). The intercept \(\beta _0'\) is then \(0\), so the regression is through the origin.
In all cases the residuals of the transformed regression should be checked again, for example with the Durbin–Watson test, to make sure the transformation has removed the autocorrelation.
Example 13.3.3. Apply the Cochrane–Orcutt method to the sales data of Example 13.2.2.
Solution. From the least-squares residuals, \(\sum _{t=2}^{15}e_te_{t-1} = 27.155\) and \(\sum _{t=2}^{15}e_{t-1}^2 = 34.287\), so \(\hat \rho = 27.155/34.287 = 0.792\). The transformed data \(y_t' = y_t - 0.792y_{t-1}\), \(x_t' = x_t - 0.792x_{t-1}\) for \(t = 2, \dots , 15\) begin \[ y_2' = 21.9 - 0.792(18.8) = 7.010 , \qquad x_2' = 5 - 0.792(4) = 1.832 , \] and the least-squares fit to the \(14\) transformed pairs is \[ \hat y' = 1.932 + 2.079x' . \] So \(\hat \beta _1 = 2.079\) and \(\hat \beta _0 = 1.932/(1 - 0.792) = 9.29\), and the fitted model is \(\hat y = 9.29 + 2.079x\), with \(\operatorname {se}(\hat \beta _1) = 0.445\) from the transformed regression (\(s^2 = 1.563\) on \(12\) degrees of freedom). The standard error is almost three times the \(0.151\) reported by ordinary least squares: the original analysis greatly overstated its precision. A \(95\%\) interval for \(\beta _1\) is now \(2.079 \pm 2.179(0.445)\), from \(1.11\) to \(3.05\).
The Durbin–Watson statistic of the transformed residuals is \(1.61\), above \(d_U = 1.35\) for \(n = 14\), so no autocorrelation remains. Iterating the method changes \(\hat \rho \) to \(0.755\) and \(\hat \beta _1\) to \(2.11\). A Hildreth–Lu grid gives:
| \(\rho \) | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 |
| Residual SS | 21.34 | 19.69 | 18.83 | 18.78 | 19.00 |
a flat minimum between \(0.7\) and \(0.8\), consistent with the other estimates.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.