2.3 Properties of the least squares estimators
The estimates \(a\) and \(b\) are computed from a sample. Another sample would give different values, and their sampling distribution is what tells us how far they may be from \(\alpha \) and \(\beta \). In this subsection all expectations are conditional on the \(X\)’s, which may therefore be treated as constants.
Lemma 2.3.1. Let \(w_i = x_i/\sum x_j^2\). Then \(\sum w_i = 0\), \(\sum w_iX_i = \sum w_ix_i = 1\), \(\sum w_i^2 = 1/\sum x_i^2\), and \[ b = \sum w_iY_i = \beta + \sum w_iu_i . \]
Proof. \(\sum w_i = \sum x_i/\sum x_j^2 = 0\) because deviations sum to zero. \(\sum w_ix_i = \sum x_i^2/\sum x_j^2 = 1\), and since \(X_i = x_i + \bar X\), \(\sum w_iX_i = \sum w_ix_i + \bar X\sum w_i = 1\). \(\sum w_i^2 = \sum x_i^2/(\sum x_j^2)^2\). Next, \(b = \sum x_iy_i/\sum x_j^2 = \sum w_i(Y_i - \bar Y) = \sum w_iY_i\). Finally, substituting \(Y_i = \alpha + \beta X_i + u_i\), \[ b = \alpha \sum w_i + \beta \sum w_iX_i + \sum w_iu_i = \beta + \sum w_iu_i . \qedhere \] □
The lemma expresses \(b\) as a linear function of the \(Y\)’s, and its sampling error \(b - \beta \) as a weighted sum of the disturbances. Everything else follows from it.
Theorem 2.3.2. Under (A1)–(A4), \begin {align*} E(b) &= \beta , & \Var (b) &= \frac {\sigma ^2}{\sum x_i^2},\\ E(a) &= \alpha , & \Var (a) &= \sigma ^2\left (\frac 1n + \frac {\bar X^2}{\sum x_i^2}\right ),\\ & & \Cov (a, b) &= -\frac {\bar X\sigma ^2}{\sum x_i^2} . \end {align*}
Proof. By Lemma 2.3.1 and (A1), \(E(b) = \beta + \sum w_iE(u_i) = \beta \). By (A2) and (A3), the variance of a weighted sum of uncorrelated terms with common variance is \[ \Var (b) = \Var \Bigl (\sum w_iu_i\Bigr ) = \sigma ^2\sum w_i^2 = \frac {\sigma ^2}{\sum x_i^2} . \] For the intercept, averaging the model gives \(\bar Y = \alpha + \beta \bar X + \bar u\), so \[ a - \alpha = \bar Y - b\bar X - \alpha = \bar u - (b - \beta )\bar X . \] Hence \(E(a) = \alpha \). Now \(\Cov (\bar u, b) = \Cov \bigl (\frac 1n\sum u_i, \sum w_iu_i\bigr ) = \frac {\sigma ^2}{n}\sum w_i = 0\), so \begin {align*} \Var (a) &= \Var (\bar u) + \bar X^2\Var (b) = \frac {\sigma ^2}{n} + \frac {\bar X^2\sigma ^2}{\sum x_i^2},\\ \Cov (a, b) &= \Cov \bigl (\bar u - (b - \beta )\bar X,\ b\bigr ) = -\bar X\Var (b) . \qedhere \end {align*} □
The formula \(\Var (b) = \sigma ^2/\sum x_i^2\) says what makes a slope precisely estimated: little noise (\(\sigma ^2\) small) and a wide spread of the regressor (\(\sum x_i^2\) large). The spread grows with the sample size, but it also depends on where the \(X\)’s are. Twenty observations bunched together tell us less about the slope than ten spread far apart. The covariance of \(a\) and \(b\) has the opposite sign to \(\bar X\): when the data lie to the right of the origin, a line that is too steep must have too low an intercept to pass near the point of means.
Example 2.3.3. Two researchers fit \(Y = \alpha + \beta x + u\) to different data sets of eight observations each:
| Researcher 1 | Researcher 2 | ||
| \(Y\) | \(x\) | \(Y\) | \(x\) |
| 4.0, 4.5, 4.5, 3.5 | 3 | 2.0, 2.5, 2.5, 1.5 | 1 |
| 4.5, 4.5, 5.5, 5.0 | 4 | 11.5, 10.5, 10.5, 11.0 | 10 |
They obtain the slope estimates \(0.750\) and \(0.972\), with standard errors \(0.339\) and \(0.038\). Why is the first standard error so much larger?
Solution. The standard error is \(s/\sqrt {\sum x_i^2}\), so either the residual scatter \(s\) or the spread \(\sum x_i^2\) must differ. Here the residual scatter is the same. With only two distinct \(x\) values the fitted line passes through the two group means, so each residual is a deviation from its group mean. In both data sets these are \((-0.125,\ 0.375,\ 0.375,\ -0.625)\) at the lower \(x\) and the values \(-0.375, -0.375, 0.625, 0.125\), in some order, at the upper \(x\). So both regressions have \(\text {RSS} = 1.375\) and \(s = \sqrt {1.375/6} = 0.479\). The spreads are very different. Researcher 1’s \(x\) takes the values \(3\) and \(4\), mean \(3.5\), so \(\sum x_i^2 = 8(0.5)^2 = 2\). Researcher 2’s takes \(1\) and \(10\), mean \(5.5\), so \(\sum x_i^2 = 8(4.5)^2 = 162\). The ratio of standard errors is therefore \[ \frac {0.479/\sqrt {2}}{0.479/\sqrt {162}} = \sqrt {81} = 9, \] and indeed \(0.339/0.038 \approx 9\). The second researcher’s slope is measured nine times as precisely, entirely because the regressor values are nine times as far apart. With the same scatter about the line, a line pinned down at two widely separated points is far less free to tilt. (The second slope is sometimes quoted as \(0.970\); least squares on these data gives \(0.9722\).)
The Gauss–Markov theorem
Least squares is not the only way to estimate \(\beta \). The slope through the first and last observations is another, as is the average of the slopes from the point of means to each observation. Why prefer least squares?
Theorem 2.3.4 (Gauss–Markov). Under (A1)–(A4), \(b\) has the smallest variance among all linear unbiased estimators of \(\beta \): if \(\tilde \beta = \sum c_iY_i\) is unbiased for \(\beta \) whatever the values of \(\alpha \) and \(\beta \), then \(\Var (\tilde \beta ) \geq \Var (b)\), with equality only if \(c_i = w_i\) for all \(i\). The corresponding statement holds for \(a\).
Proof. \(E(\tilde \beta ) = \sum c_i(\alpha + \beta X_i) = \alpha \sum c_i + \beta \sum c_iX_i\). For this to equal \(\beta \) for all \(\alpha \) and \(\beta \) we need \[ \sum c_i = 0 \qquad \text {and}\qquad \sum c_iX_i = 1 . \] Then \(\tilde \beta = \beta + \sum c_iu_i\) and \(\Var (\tilde \beta ) = \sigma ^2\sum c_i^2\). Write \(c_i = w_i + d_i\). Both \(c\) and \(w\) satisfy the two conditions, so \(\sum d_i = 0\) and \(\sum d_iX_i = 0\), and therefore \[ \sum w_id_i = \frac {\sum x_id_i}{\sum x_j^2} = \frac {\sum X_id_i - \bar X\sum d_i}{\sum x_j^2} = 0 . \] Hence \[ \Var (\tilde \beta ) = \sigma ^2\Bigl (\sum w_i^2 + 2\sum w_id_i + \sum d_i^2\Bigr ) = \Var (b) + \sigma ^2\sum d_i^2 \geq \Var (b), \] with equality only if every \(d_i = 0\). The argument for \(a\) is identical, with the unbiasedness conditions \(\sum c_i = 1\) and \(\sum c_iX_i = 0\). □
Least squares is therefore BLUE, the best linear unbiased estimator. The theorem needs no normality. Its limits matter as much as its content. It compares only linear and unbiased estimators, so a biased or non-linear estimator may do better: ridge regression, for instance, trades a little bias for a large fall in variance. And it relies on (A2) and (A3): when the disturbances are heteroscedastic or autocorrelated, least squares is still linear and unbiased but no longer best, and generalised least squares takes its place.
Estimating \(\sigma ^2\)
The variances in Theorem 2.3.2 involve the unknown \(\sigma ^2\). The natural estimate uses the residuals, but the divisor must be \(n - 2\), not \(n\).
Theorem 2.3.5. Under (A1)–(A4), \(E\bigl (\sum e_i^2\bigr ) = (n-2)\sigma ^2\), so \[ s^2 = \frac {\sum e_i^2}{n-2} \] is an unbiased estimator of \(\sigma ^2\).
Proof. From \(y_i = \beta x_i + (u_i - \bar u)\) and \(\hat y_i = bx_i\), the residual is \[ e_i = y_i - bx_i = (u_i - \bar u) - (b - \beta )x_i . \] Squaring and summing, \[ \sum e_i^2 = \sum (u_i - \bar u)^2 - 2(b - \beta )\sum x_i(u_i - \bar u) + (b - \beta )^2\sum x_i^2 . \] Now \(\sum x_i(u_i - \bar u) = \sum x_iu_i = (b - \beta )\sum x_i^2\) by Lemma 2.3.1, so the last two terms combine to \(-(b - \beta )^2\sum x_i^2\). Taking expectations, \(E\sum (u_i - \bar u)^2 = (n-1)\sigma ^2\), the familiar result for a sample variance, and \(E\bigl [(b - \beta )^2\bigr ]\sum x_i^2 = \Var (b)\sum x_i^2 = \sigma ^2\). Hence \(E\sum e_i^2 = (n-1)\sigma ^2 - \sigma ^2 = (n-2)\sigma ^2\). □
The residuals are not \(n\) free numbers: they satisfy the two linear restrictions \(\sum e_i = 0\) and \(\sum x_ie_i = 0\). Two degrees of freedom have been used to estimate \(\alpha \) and \(\beta \), and the divisor \(n - 2\) is the number that remain. The estimated standard errors are \[ \operatorname {se}(b) = \frac {s}{\sqrt {\sum x_i^2}}, \qquad \operatorname {se}(a) = s\sqrt {\frac 1n + \frac {\bar X^2}{\sum x_i^2}} . \]
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.