[next] [prev] [prev-tail] [tail] [up]

11.4 Estimation with autocorrelated disturbances

Generalised least squares

With \(\rho \) known, Theorem 11.1.2 and Aitken’s Theorem 9.4.2 say that the best linear unbiased estimator is least squares on the transformed data \begin {equation} \label {eq:praiswinsten} Y_1^* = \sqrt {1 - \rho ^2}\,Y_1,\quad x_1^* = \sqrt {1 - \rho ^2}\,x_1; \qquad Y_t^* = Y_t - \rho Y_{t-1},\quad x_t^* = x_t - \rho x_{t-1} \quad (t \geq 2) . \end {equation} This is the Prais–Winsten transformation. Each observation is quasi-differenced, and the first is rescaled so that its disturbance has the same variance as the others. Note that the intercept column becomes \(\sqrt {1 - \rho ^2}\) followed by \(1 - \rho \) repeated, so the transformed regression has no constant term in the ordinary sense. The older Cochrane–Orcutt transformation simply drops the first observation. It loses little when \(n\) is large and the regressors are stationary, but in short samples, and particularly with trending regressors, the first observation can carry a great deal of information, and dropping it can make the estimator worse even than least squares.

Feasible GLS

In practice \(\rho \) is unknown. The standard procedure is:

1.
Estimate the model by least squares and compute \(\hat \rho = \sum e_te_{t-1}/\sum e_{t-1}^2\) from the residuals.
2.
Apply the Prais–Winsten (or Cochrane–Orcutt) transformation with \(\hat \rho \) and re-estimate \(\beta \).
3.
Optionally recompute the residuals \(Y_t - x_t'\hat \beta \) with the new estimate, re-estimate \(\rho \), and repeat until the estimates settle.

The iterated procedure converges to a stationary point of the sum of squares in the transformed model; since the sum of squares can have more than one local minimum, Hildreth and Lu recommended a grid search over \(\rho \in (-1, 1)\) instead. Durbin’s two-step method estimates \(\rho \) differently, as the coefficient of \(Y_{t-1}\) in the regression of \(Y_t\) on \(Y_{t-1}\), \(x_t\) and \(x_{t-1}\), which is justified by Theorem 11.4.2 below. Under standard conditions every one of these feasible estimators has the same asymptotic distribution as GLS with \(\rho \) known.

Theorem 11.4.1. If \(u_t\) is a stationary AR(1) scheme with normal \(\varepsilon _t\), the exact log-likelihood is \[ \ell (\beta , \rho , \sigma _\varepsilon ^2) = -\frac n2\log (2\pi \sigma _\varepsilon ^2) + \frac 12\log (1 - \rho ^2) - \frac {1}{2\sigma _\varepsilon ^2}(PY - PX\beta )'(PY - PX\beta ) . \] For given \(\rho \) it is maximised by the Prais–Winsten estimator, with \(\hat \sigma _\varepsilon ^2(\rho ) = \text {RSS}(\rho )/n\), and the concentrated log-likelihood is \[ \ell ^*(\rho ) = \text {const} - \frac n2\log \text {RSS}(\rho ) + \frac 12\log (1 - \rho ^2) . \]

Proof. By Theorem 9.4.2(iv) the log density of \(Y\) is \(-\frac n2\log (2\pi ) - \frac 12\log |\sigma _\varepsilon ^2\Omega | - \frac {1}{2\sigma _\varepsilon ^2}(Y - X\beta )'\Omega ^{-1}(Y - X\beta )\). By Theorem 11.1.2, \(|\sigma _\varepsilon ^2\Omega | = \sigma _\varepsilon ^{2n}/(1 - \rho ^2)\) and \(\Omega ^{-1} = P'P\), which gives \(\ell \). For fixed \(\rho \), maximising over \(\beta \) minimises \(\text {RSS}(\rho ) = \|PY - PX\beta \|^2\), which is least squares on the transformed data; then \(\partial \ell /\partial \sigma _\varepsilon ^2 = 0\) gives \(\hat \sigma _\varepsilon ^2 = \text {RSS}(\rho )/n\), and substituting back gives \(\ell ^*\). □

Maximum likelihood is therefore Hildreth–Lu with a penalty: it minimises \(\text {RSS}(\rho )(1 - \rho ^2)^{-1/n}\) rather than \(\text {RSS}(\rho )\). The term \(\frac 12\log (1 - \rho ^2)\) tends to \(-\infty \) as \(|\rho | \to 1\), so the maximum likelihood estimate is always inside the stationary region, as in Theorem 4.6.1(ii).

Autocorrelation as a symptom: common factors

Substituting \(u_t = Y_t - x_t'\beta \) into \(u_t = \rho u_{t-1} + \varepsilon _t\) shows what the AR(1) error model really assumes.

Theorem 11.4.2. The model \(Y_t = x_t'\beta + u_t\), \(u_t = \rho u_{t-1} + \varepsilon _t\) is identical to \[ Y_t = \rho Y_{t-1} + x_t'\beta - \rho \,x_{t-1}'\beta + \varepsilon _t , \] which is the autoregressive distributed lag model \[ Y_t = \alpha Y_{t-1} + x_t'\gamma _0 + x_{t-1}'\gamma _1 + \varepsilon _t \] subject to the restrictions \(\gamma _1 = -\alpha \gamma _0\), one for each regressor other than the intercept. In lag-operator form both sides of \((1 - \rho L)Y_t = (1 - \rho L)x_t'\beta + \varepsilon _t\) share the common factor \(1 - \rho L\).

Proof. \(u_t - \rho u_{t-1} = (Y_t - x_t'\beta ) - \rho (Y_{t-1} - x_{t-1}'\beta ) = \varepsilon _t\); rearrange. Matching coefficients with the ADL model gives \(\alpha = \rho \), \(\gamma _0 = \beta \) and \(\gamma _1 = -\rho \beta = -\alpha \gamma _0\). For the intercept, \(x_t = x_{t-1} = 1\) and the two terms merge into \((1 - \rho )\beta _1\), so no restriction arises. □

The restrictions are testable: estimate the unrestricted ADL by least squares, the restricted model by maximum likelihood, and compare them with a likelihood ratio test (or test \(\gamma _1 + \alpha \gamma _0 = 0\) by the Wald test of the delta method). Hendry and Mizon (1978) argued that this is the right order of thought. A significant Durbin–Watson or Breusch–Godfrey statistic says that the static model is inadequate; it does not say that the inadequacy takes the special form of AR(1) disturbances. Often the dynamics are richer, and correcting for autocorrelation mechanically, by Cochrane–Orcutt or Prais–Winsten, imposes restrictions that the data reject. In their phrase, autocorrelation should be treated as a convenient simplification to be tested, not a nuisance to be removed.

Heteroscedasticity and autocorrelation consistent standard errors

The alternative to modelling the autocorrelation is to keep least squares and correct the standard errors, as White’s estimator does for heteroscedasticity. With \(\Var (U) = \Sigma \) of unknown form, \[ \Var (b) = (X'X)^{-1}X'\Sigma X(X'X)^{-1}, \qquad X'\Sigma X = \sum _t\sum _s\Cov (u_t, u_s)\,x_tx_s' . \] Under stationarity, \(\frac 1nX'\Sigma X\) converges to \(S = \sum _{j=-\infty }^{\infty }\Gamma _j\), where \(\Gamma _j = E(u_tu_{t-j}x_tx_{t-j}')\) and \(\Gamma _{-j} = \Gamma _j'\). Let \(g_t = e_tx_t\) and \(\hat \Gamma _j = \sum _{t=j+1}^ng_tg_{t-j}'\). Simply adding the \(\hat \Gamma _j\) up to a lag \(L\) can give a matrix that is not positive semi-definite, and hence negative “variances”. Newey and West’s solution is to down-weight the higher lags.

Theorem 11.4.3 (Newey–West). For any lag \(L \geq 0\), the matrix \[ \hat S_L = \hat \Gamma _0 + \sum _{j=1}^L\Bigl (1 - \frac {j}{L + 1}\Bigr )\bigl (\hat \Gamma _j + \hat \Gamma _j'\bigr ) \] is positive semi-definite, and so therefore is \(\widehat \Var _{\text {NW}}(b) = (X'X)^{-1}\hat S_L(X'X)^{-1}\). If \(L \to \infty \) with \(L/n^{1/4} \to 0\), and under suitable conditions on the dependence and moments, \(\widehat \Var _{\text {NW}}(b)\) is consistent in the sense of Theorem 10.2.1, whatever the form of the heteroscedasticity and autocorrelation.

Proof of positive semi-definiteness. Set \(g_t = 0\) for \(t < 1\) and \(t > n\), so that \(\hat \Gamma _j = \sum _tg_tg_{t-j}'\) with the sum over all integers. For each \(t\) define the moving sum \(a_t = g_t + g_{t+1} + \dots + g_{t+L}\). Then \[ \sum _ta_ta_t' = \sum _{i=0}^L\sum _{i'=0}^L\sum _tg_{t+i}g_{t+i'}' = \sum _{i=0}^L\sum _{i'=0}^L\hat \Gamma _{i - i'}, \] with \(\hat \Gamma _{-j} = \hat \Gamma _j'\). The number of pairs \((i, i')\) with \(i - i' = j\) is \(L + 1 - |j|\), so \[ \sum _ta_ta_t' = (L + 1)\hat \Gamma _0 + \sum _{j=1}^L(L + 1 - j)(\hat \Gamma _j + \hat \Gamma _j') = (L + 1)\hat S_L . \] A sum of matrices \(a_ta_t'\) is positive semi-definite, and so is \(\hat S_L\). Pre- and post-multiplying by the symmetric matrix \((X'X)^{-1}\) preserves this. □

The weights \(1 - j/(L + 1)\) are called Bartlett weights, and the proof shows where they come from: they are what results from estimating the variance of moving sums. \(L = 0\) gives White’s estimator. The choice of \(L\) is a trade-off: too small and autocorrelation beyond lag \(L\) is ignored; too large and the estimate is noisy. A common rule is \(L = \lfloor 4(n/100)^{2/9}\rfloor \), which gives \(L = 4\) for \(n\) between about \(100\) and \(270\). Newey–West standard errors, also called HAC standard errors, are standard in finance and in macroeconomic work where the model is believed to be correct in the mean but its disturbances are serially correlated, for example regressions on overlapping multi-period returns.

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.