13.4 Model selection criteria
Often several models pass the checks, and one must be chosen. The information criteria balance fit against complexity: \[ \text {AIC} = -2\log L + 2r , \qquad \text {BIC} = -2\log L + r\log n , \] where \(L\) is the maximised likelihood, \(r\) the number of estimated ARMA parameters and \(n\) the length of the series. The model with the smallest value is preferred.
Proposition 13.4.1. For Gaussian shocks, using the conditional likelihood, \[ -2\log L = n\log \hat \sigma _Z^2 + n(1 + \log 2\pi ) , \] where \(\hat \sigma _Z^2 = S/n\) is the residual mean square. So, up to a constant common to all models, \[ \text {AIC} = n\log \hat \sigma _Z^2 + 2r , \qquad \text {BIC} = n\log \hat \sigma _Z^2 + r\log n . \]
Proof. Given the parameters, the reconstructed shocks \(z_t\) are independent \(N(0, \sigma _Z^2)\), so \[ \log L = -\frac n2\log (2\pi \sigma _Z^2) - \frac {S}{2\sigma _Z^2} , \qquad S = \sum _tz_t^2 . \] For fixed parameters this is largest at \(\sigma _Z^2 = S/n\), where \(S/(2\sigma _Z^2) = n/2\), so \(\log L = -\frac n2(\log 2\pi + \log \hat \sigma _Z^2 + 1)\). Multiplying by \(-2\) gives the result. □
Adding parameters always reduces \(\hat \sigma _Z^2\); the penalty asks whether the reduction is worth it. BIC’s penalty, \(\log n\) per parameter, exceeds AIC’s \(2\) once \(n \ge 8\), so BIC tends to choose smaller models. AIC aims at good prediction and tends to overfit slightly in large samples; BIC is consistent, choosing the true order with probability tending to \(1\) when the true model is among the candidates. For forecasting, a slightly smaller model is usually the safer error.
Example 13.4.2. For the series of Example 11.2.2 (\(n = 150\)), several models were fitted by conditional least squares. The table gives the residual mean square, AIC and BIC (as in Proposition 13.4.1), and the Ljung–Box statistic with \(K = 10\):
| Model | \(\hat \sigma _Z^2\) | AIC | BIC | \(Q^*\) | df | \(p\)-value |
| AR(1) | 1.1034 | 16.76 | 19.77 | 29.67 | 9 | 0.000 |
| AR(2) | 0.9299 | \(-6.90\) | \(-0.88\) | 7.76 | 8 | 0.457 |
| AR(3) | 0.9205 | \(-6.43\) | 2.60 | 4.36 | 7 | 0.737 |
| MA(2) | 0.9401 | \(-5.26\) | 0.76 | 7.79 | 8 | 0.455 |
| ARMA(1,1) | 0.9449 | \(-4.50\) | 1.52 | 7.96 | 8 | 0.438 |
| ARMA(2,1) | 0.9155 | \(-7.25\) | 1.78 | 3.85 | 7 | 0.797 |
The residual autocorrelations of the AR(2) fit, for \(k = 1, \dots , 10\), were \(0.047\), \(-0.059\), \(0.089\), \(-0.004\), \(-0.043\), \(-0.027\), \(-0.101\), \(-0.113\), \(-0.087\), \(-0.047\), and those of the AR(1) fit began \(0.159\), \(-0.351\), \(-0.153\). Which model should be used?
Solution. AR(1) is inadequate. Its residual \(r_2 = -0.351\) lies far outside \(\pm 2/\sqrt {150} = \pm 0.163\): the model has missed the lag-2 dependence that the partial ac.f. showed. Its \(Q^* = 29.67\) exceeds \(\chi ^2_{0.05,9} = 16.92\) (the \(p\)-value is essentially \(0\)).
AR(2) passes. All ten residual autocorrelations lie within \(\pm 0.163\). For the Ljung–Box statistic, \begin {align*} Q^* &= 150(152)\Bigl [\frac {0.047^2}{149} + \frac {0.059^2}{148} + \dots + \frac {0.047^2}{140}\Bigr ] \\ &= 22800 \times 0.000341 = 7.77 , \end {align*}
on \(10 - 2 = 8\) degrees of freedom, well below \(\chi ^2_{0.05,8} = 15.51\) (the table’s \(7.76\) uses unrounded \(r_k\); the \(p\)-value is \(0.46\)). The residuals are consistent with white noise. Overfitting confirms it: the AR(3) estimate of \(\alpha _3\) is \(0.11\), with standard error about \(1/\sqrt {150} = 0.08\), so it is not significant, and \(\hat \alpha _1\), \(\hat \alpha _2\) change only from \((0.554, -0.402)\) to \((0.601, -0.466)\).
Choosing among the adequate models. BIC is smallest for AR(2), by a clear margin of \(1.6\) over MA(2). AIC is smallest for ARMA(2,1), but only by \(0.35\) below AR(2), a negligible difference bought with an extra parameter. Both criteria prefer AR(2) to the MA(2) and ARMA(1,1) alternatives. The AR(2) model, \(\hat x_t = 0.554x_{t-1} - 0.402x_{t-2}\) with \(\hat \sigma _Z^2 = 0.93\), is the sensible choice: simplest, adequate, and preferred by BIC. (The data were simulated from an AR(2).)
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.