12.2 Fitting an autoregressive model
Suppose we have an AR(\(p\)) process with mean \(\mu \), \[ X_t - \mu = \alpha _1(X_{t-1} - \mu ) + \alpha _2(X_{t-2} - \mu ) + \dots + \alpha _p(X_{t-p} - \mu ) + Z_t . \] If \(x_1, x_2, \dots , x_n\) are the observations, the parameters \(\mu , \alpha _1, \dots , \alpha _p\) can be estimated by least squares, minimising \[ S = \sum _{t=p+1}^n\bigl [x_t - \mu - \alpha _1(x_{t-1} - \mu ) - \dots - \alpha _p(x_{t-p} - \mu )\bigr ]^2 . \] In practice \(\mu \) is estimated by \(\bar x\), and then \(S\) is a multiple regression of \(x_t - \bar x\) on its own lagged values, so the usual least-squares formulas apply directly. (Because the first \(p\) observations serve only as regressors, this is called conditional least squares.) An alternative is the Yule–Walker estimator: replace \(\rho (k)\) by \(r_k\) in the equations of Theorem 8.2.1 and solve, \[ \hat {\boldsymbol \alpha } = \hat {\mathbf {R}}_p^{-1}\hat {\boldsymbol \rho }_p , \qquad \hat \sigma _Z^2 = c_0\bigl (1 - \hat \alpha _1r_1 - \dots - \hat \alpha _pr_p\bigr ) , \] where \(\hat {\mathbf {R}}_p\) has entries \(r_{|i-j|}\) and \(\hat {\boldsymbol \rho }_p = (r_1, \dots , r_p)^\top \). The two methods are close for long series.
Proposition 12.2.1. Take \(\mu = 0\) for simplicity. The normal equations of conditional least squares are \[ \sum _{t=p+1}^nx_tx_{t-j} = \sum _{i=1}^p\hat \alpha _i\sum _{t=p+1}^nx_{t-i}x_{t-j} , \qquad j = 1, \dots , p , \] and each sum of lagged products differs from \(nc_{|i-j|}\) or \(nc_j\) by at most \(p\) end terms. So for large \(n\) the least-squares and Yule–Walker estimates agree.
Proof. Differentiating \(S = \sum _t(x_t - \sum _i\alpha _ix_{t-i})^2\) with respect to \(\alpha _j\) and setting the result to zero gives the stated equations, the normal equations of the regression of \(x_t\) on \(x_{t-1}, \dots , x_{t-p}\). Each sum \(\sum _{t=p+1}^nx_{t-i}x_{t-j}\) is a sum of products of observations \(|i - j|\) apart; it contains the same terms as \(nc_{|i-j|} = \sum _{s=1}^{n-|i-j|}x_sx_{s+|i-j|}\) except for at most \(p\) at the ends. Dividing both sides by \(nc_0\), the equations become those of Theorem 8.2.1 with \(r\)’s in place of \(\rho \)’s, up to terms of order \(p/n\). □
Proposition 12.2.2. For an AR(1) process, the least-squares estimate \[ \hat \alpha = \frac {\sum _{t=2}^nx_tx_{t-1}}{\sum _{t=2}^nx_{t-1}^2} \] has, for large \(n\), approximate variance \[ \Var (\hat \alpha ) \approx \frac {1 - \alpha ^2}{n} . \]
Derivation. Treating the regressors \(x_{t-1}\) as fixed, regression theory gives \(\Var (\hat \alpha ) = \sigma _Z^2/\sum _tx_{t-1}^2\). For large \(n\), \(\sum _tx_{t-1}^2 \approx n\sigma _X^2 = n\sigma _Z^2/(1 - \alpha ^2)\) (Theorem 8.1.1), and the ratio is \((1 - \alpha ^2)/n\). (A careful argument, with the regressors random, gives the same large-sample result.) □
Note that the precision is greatest when \(|\alpha |\) is close to \(1\): a strongly autocorrelated series pins down its own autocorrelation well. The order \(p\) is chosen as in Section 11, from the sample partial ac.f. (whose value at lag \(p\) is exactly \(\hat \alpha _p\) in the Yule–Walker fit) or by comparing AIC across orders.
Example 12.2.3. Ten observations from a process with known mean \(0\) are \[ 0, \ -2, \ -4, \ -2, \ 0, \ 2, \ 2, \ 1, \ 2, \ 1 . \] Fit an AR(1) model by the Yule–Walker and least-squares methods, and give an approximate standard error.
Solution. The sum of squares is \(\sum x_t^2 = 0 + 4 + 16 + 4 + 0 + 4 + 4 + 1 + 4 + 1 = 38\), and the lagged products \(x_tx_{t-1}\) for \(t = 2, \dots , 10\) are \(0\), \(8\), \(8\), \(0\), \(0\), \(4\), \(2\), \(2\), \(2\), adding to \(26\).
Yule–Walker. With the known mean, \(c_0 = 38/10 = 3.8\) and \(c_1 = 26/10 = 2.6\), so \(\hat \alpha = r_1 = 26/38 = 0.684\), and \(\hat \sigma _Z^2 = c_0(1 - r_1^2) = 3.8(1 - 0.468) = 2.02\).
Least squares. The denominator omits the last observation: \(\sum _{t=2}^{10}x_{t-1}^2 = 38 - 1^2 = 37\), so \(\hat \alpha = 26/37 = 0.703\). The residuals \(x_t - 0.703x_{t-1}\) are \(-2.00\), \(-2.59\), \(0.81\), \(1.41\), \(2.00\), \(0.59\), \(-0.41\), \(1.30\), \(-0.41\), with sum of squares \(19.73\); on \(9 - 1 = 8\) degrees of freedom, \(\hat \sigma _Z^2 = 2.47\).
The two estimates of \(\alpha \) differ by \(0.02\). That is small compared with the standard error from Proposition 12.2.2, which is \(\sqrt {(1 - 0.703^2)/10} = 0.22\). With only ten observations, \(\alpha \) is known only roughly: an approximate \(95\%\) interval is \(0.70 \pm 0.44\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.