4.4 Least squares in large samples
Suppose now that \((X_i, Y_i)\), \(i = 1, \dots , n\), are independent draws from a population in which \[ Y_i = \alpha + \beta X_i + u_i, \qquad E(u_i) = 0, \qquad \Cov (X_i, u_i) = 0 , \] with finite fourth moments. This is far weaker than (A1): nothing is said about the conditional mean of \(u\), only that it is uncorrelated with \(X\).
Theorem 4.4.1. Under these conditions, with \(\Var (X) > 0\):
- (i)
- \(b \xrightarrow {p} \beta \): least squares is consistent;
- (ii)
- if also \(E(u_i^2 \mid X_i) = \sigma ^2\), then \(\sqrt n(b - \beta ) \xrightarrow {d} N\bigl (0, \sigma ^2/\Var (X)\bigr )\);
- (iii)
- without that condition, \(\sqrt n(b - \beta ) \xrightarrow {d} N\Bigl (0,\ E\bigl [(X - \mu _X)^2u^2\bigr ]\big /\Var (X)^2\Bigr )\).
Proof. Write \[ b - \beta = \frac {\frac 1n\sum (X_i - \bar X)u_i}{\frac 1n\sum (X_i - \bar X)^2} . \] The denominator is \(\frac 1n\sum (X_i - \mu _X)^2 - (\bar X - \mu _X)^2 \xrightarrow {p} \Var (X)\) by the law of large numbers and Theorem 4.2.5. Likewise the numerator is \(\frac 1n\sum (X_i - \mu _X)u_i - (\bar X - \mu _X)\bar u \xrightarrow {p} \Cov (X, u) - 0 = 0\). So \(b - \beta \xrightarrow {p} 0/\Var (X) = 0\), which is (i).
For (ii) and (iii), multiply the numerator by \(\sqrt n\): \[ \frac {1}{\sqrt n}\sum (X_i - \mu _X)u_i - \sqrt n(\bar X - \mu _X)\,\bar u . \] The terms \((X_i - \mu _X)u_i\) are independent and identically distributed with mean \(\Cov (X, u) = 0\) and variance \(E\bigl [(X - \mu _X)^2u^2\bigr ]\), so the first term converges in distribution to \(N\bigl (0, E[(X - \mu _X)^2u^2]\bigr )\) by the CLT. In the second, \(\sqrt n(\bar X - \mu _X)\) converges in distribution and \(\bar u \xrightarrow {p} 0\), so the product converges to zero by Slutsky. Dividing by the denominator, which converges in probability to \(\Var (X)\), and applying Slutsky again gives (iii). Under \(E(u^2 \mid X) = \sigma ^2\), iterated expectations give \(E\bigl [(X - \mu _X)^2u^2\bigr ] = \sigma ^2\Var (X)\), and (iii) reduces to (ii). □
Three lessons follow. First, consistency needs only zero correlation between regressor and disturbance; zero conditional mean was needed for unbiasedness. Second, the large-sample variance in (ii) is \(\sigma ^2/(n\Var (X))\), which is what \(\sigma ^2/\sum x_i^2\) estimates, so the usual standard error and the usual \(t\) test are justified in large samples without normality: \(t\) is then approximately \(N(0, 1)\) rather than exactly \(t_{n-2}\). Third, when the variance of \(u\) depends on \(X\), (iii) shows that the usual formula is wrong. The correct “sandwich” variance is the basis of the heteroscedasticity-robust standard errors of the section on heteroscedasticity.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.