9.4 Non-spherical disturbances and generalised least squares

Suppose now that \[ Y = X\beta + U, \qquad E(U) = 0, \qquad \Var (U) = \sigma ^2\Omega , \] where \(\Omega \) is a known positive definite matrix, no longer the identity. Heteroscedasticity makes \(\Omega \) diagonal with unequal elements; autocorrelation fills its off-diagonal elements. The disturbances are then called non-spherical.

Theorem 9.4.1. With \(\Var (U) = \sigma ^2\Omega \), least squares remains unbiased, but \[ \Var (b) = \sigma ^2(X'X)^{-1}X'\Omega X(X'X)^{-1} , \] which in general differs from \(\sigma ^2(X'X)^{-1}\). The conventional standard errors are therefore wrong, and \(b\) is not the best linear unbiased estimator.

Proof. \(b - \beta = (X'X)^{-1}X'U\) has mean zero and variance \((X'X)^{-1}X'(\sigma ^2\Omega )X(X'X)^{-1}\). That \(b\) is not best follows from the next theorem, which exhibits an estimator with smaller variance. □

The remedy uses Corollary 5.1.4(iii): there is a non-singular matrix \(P\) with \(P'P = \Omega ^{-1}\), and then \(P\Omega P' = I\).

Theorem 9.4.2 (Aitken). Let \(P\) be non-singular with \(P'P = \Omega ^{-1}\), and transform the model to \[ Y_* = X_*\beta + U_*, \qquad Y_* = PY,\quad X_* = PX,\quad U_* = PU . \]

(i): \(\Cov (PU) = \sigma ^2I\): the transformed model satisfies the classical assumptions.
(ii): Least squares on the transformed model gives the generalised least squares (GLS) estimator \[ b_{\text {GLS}} = (X_*'X_*)^{-1}X_*'Y_* = (X'\Omega ^{-1}X)^{-1}X'\Omega ^{-1}Y , \] with \(\Var (b_{\text {GLS}}) = \sigma ^2(X'\Omega ^{-1}X)^{-1}\).
(iii): \(b_{\text {GLS}}\) is the best linear unbiased estimator of \(\beta \), and in particular \(\Var (b) - \Var (b_{\text {GLS}})\) is psd.
(iv): If \(U \sim N(0, \sigma ^2\Omega )\), \(b_{\text {GLS}}\) is the maximum likelihood estimator of \(\beta \).

Proof. (i) \(\Cov (PU) = P(\sigma ^2\Omega )P' = \sigma ^2P\Omega P' = \sigma ^2I\), by Corollary 5.1.4(iii).

(ii) \(X_*'X_* = X'P'PX = X'\Omega ^{-1}X\) and \(X_*'Y_* = X'\Omega ^{-1}Y\). \(X_*\) has rank \(k\) because \(P\) is non-singular. The variance is \(\sigma ^2(X_*'X_*)^{-1}\) by Theorem 7.1.1, applied to the transformed model.

(iii) The transformed model satisfies (A1)–(A4), so by the Gauss–Markov theorem its least squares estimator is best among estimators linear in \(Y_*\) and unbiased. Every estimator linear in \(Y\) is linear in \(Y_* = PY\), since \(AY = (AP^{-1})Y_*\), so the class is the same. Least squares on the original model is one such estimator.

(iv) The density of \(Y\) is \[ (2\pi \sigma ^2)^{-n/2}|\Omega |^{-1/2}\exp \Bigl \{-\frac {1}{2\sigma ^2}(y - X\beta )'\Omega ^{-1}(y - X\beta )\Bigr \} , \] using \(|\sigma ^2\Omega | = \sigma ^{2n}|\Omega |\). For fixed \(\sigma ^2\) it is maximised over \(\beta \) by minimising \((y - X\beta )'\Omega ^{-1}(y - X\beta ) = (y_* - X_*\beta )'(y_* - X_*\beta )\), whose minimiser is \(b_{\text {GLS}}\). □

GLS weights the observations according to how informative they are. In the commonest case, \(\Omega = \operatorname {diag}(\omega _1, \dots , \omega _n)\), one can take \(P = \operatorname {diag}(\omega _i^{-1/2})\), so each observation is divided by its own standard deviation (up to \(\sigma \)): observations with large variance are down-weighted. This is weighted least squares. With autocorrelated disturbances, \(P\) instead takes quasi-differences of successive observations, as in the Prais–Winsten transformation of the section on autocorrelation.

In practice \(\Omega \) is never known. It is modelled as a function of a few parameters, for example \(\omega _i = X_i^\gamma \) or an AR(1) structure, those parameters are estimated, usually from least squares residuals, and \(\hat \Omega \) is used in place of \(\Omega \). This feasible GLS estimator is no longer exactly linear or unbiased, but under suitable conditions it has the same asymptotic distribution as GLS with known \(\Omega \). The next two sections develop it for heteroscedasticity and for autocorrelation.

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.