2.4 Properties of the least-squares estimators
The estimates \(\hat \beta _0\) and \(\hat \beta _1\) depend on the \(y_i\), which are random; a new sample at the same \(x_i\) would give different estimates. So \(\hat \beta _0\) and \(\hat \beta _1\) are random variables, and we ask for their means and variances. Throughout, A1–A3 hold.
Lemma 2.4.1. \(\hat \beta _1\) is a linear combination of the \(y_i\): \[ \hat \beta _1 = \sum _{i=1}^n c_i y_i, \qquad \text {where } c_i = \frac {x_i - \bar x}{S_{xx}} , \] and the constants \(c_i\) satisfy \[ \sum c_i = 0, \qquad \sum c_i x_i = 1, \qquad \sum c_i^2 = \frac {1}{S_{xx}} . \]
Proof. By Lemma 2.2.3(iii), \(\hat \beta _1 = S_{xy}/S_{xx} = \sum (x_i - \bar x) y_i / S_{xx} = \sum c_i y_i\). Next, \(\sum c_i = \sum (x_i - \bar x)/S_{xx} = 0\) by Lemma 2.2.3(i). For the second, \[ \sum c_i x_i = \frac {\sum (x_i - \bar x) x_i}{S_{xx}} = \frac {S_{xx}}{S_{xx}} = 1 , \] using Lemma 2.2.3(iii) with \(y\) replaced by \(x\), which says \(\sum (x_i - \bar x) x_i = S_{xx}\). Finally \(\sum c_i^2 = \sum (x_i - \bar x)^2 / S_{xx}^2 = S_{xx}/S_{xx}^2 = 1/S_{xx}\). □
Theorem 2.4.2. \(\hat \beta _1\) is an unbiased estimator of \(\beta _1\), with variance \[ E(\hat \beta _1) = \beta _1, \qquad \operatorname {Var}(\hat \beta _1) = \frac {\sigma ^2}{S_{xx}} . \]
Proof. By Lemma 2.4.1 and Theorem 2.1.1, \begin {align*} E(\hat \beta _1) &= \sum c_i E(y_i) = \sum c_i (\beta _0 + \beta _1 x_i) \\ &= \beta _0 \sum c_i + \beta _1 \sum c_i x_i = \beta _0 \cdot 0 + \beta _1 \cdot 1 = \beta _1 . \end {align*}
The \(y_i\) are uncorrelated with common variance \(\sigma ^2\), so the variance of a linear combination is the sum of the squared coefficients times \(\sigma ^2\): \[ \operatorname {Var}(\hat \beta _1) = \sum c_i^2 \operatorname {Var}(y_i) = \sigma ^2 \sum c_i^2 = \frac {\sigma ^2}{S_{xx}} . \qedhere \] □
Remark. \(\operatorname {Var}(\hat \beta _1)\) is small when \(S_{xx}\) is large, that is, when the \(x_i\) are spread widely. When we can choose the \(x\) values, as with advertising spend, spreading them out gives a more precise slope.
Proof. \(\bar y = \sum \frac 1n y_i\) and \(\hat \beta _1 = \sum c_i y_i\). As the \(y_i\) are uncorrelated with variance \(\sigma ^2\), \[ \operatorname {Cov}\Bigl (\sum \tfrac 1n y_i, \sum c_j y_j\Bigr ) = \sum _i \tfrac 1n c_i \operatorname {Var}(y_i) = \frac {\sigma ^2}{n} \sum c_i = 0 . \qedhere \] □
Theorem 2.4.4. \(\hat \beta _0\) is an unbiased estimator of \(\beta _0\), and \[ \operatorname {Var}(\hat \beta _0) = \sigma ^2 \left ( \frac 1n + \frac {\bar x^2}{S_{xx}} \right ), \qquad \operatorname {Cov}(\hat \beta _0, \hat \beta _1) = -\frac {\bar x \sigma ^2}{S_{xx}} . \]
Proof. \(E(\bar y) = \frac 1n \sum (\beta _0 + \beta _1 x_i) = \beta _0 + \beta _1 \bar x\), so by Theorem 2.4.2 \[ E(\hat \beta _0) = E(\bar y) - \bar x E(\hat \beta _1) = \beta _0 + \beta _1 \bar x - \bar x \beta _1 = \beta _0 . \] For the variance, \(\operatorname {Var}(\bar y) = \sigma ^2/n\), and by Lemma 2.4.3 the covariance term vanishes: \[ \operatorname {Var}(\hat \beta _0) = \operatorname {Var}(\bar y) + \bar x^2 \operatorname {Var}(\hat \beta _1) - 2\bar x \operatorname {Cov}(\bar y, \hat \beta _1) = \frac {\sigma ^2}{n} + \frac {\bar x^2 \sigma ^2}{S_{xx}} . \] Finally, again using Lemma 2.4.3, \begin {align*} \operatorname {Cov}(\hat \beta _0, \hat \beta _1) &= \operatorname {Cov}(\bar y - \bar x \hat \beta _1, \hat \beta _1) = \operatorname {Cov}(\bar y, \hat \beta _1) - \bar x \operatorname {Var}(\hat \beta _1) \\ &= -\frac {\bar x \sigma ^2}{S_{xx}} . \qedhere \end {align*} □
When \(\bar x > 0\) the covariance is negative. That matches the picture: the line pivots about \((\bar x, \bar y)\), so a sample that gives too steep a slope tends to give too low an intercept.
The least-squares estimators are not just unbiased; among all unbiased estimators that are linear in the \(y_i\), they have the smallest variance.
Theorem 2.4.5 (Gauss–Markov, for the slope). Under A1–A3, let \(\tilde \beta _1 = \sum k_i y_i\) be any linear unbiased estimator of \(\beta _1\). Then \(\operatorname {Var}(\tilde \beta _1) \ge \operatorname {Var}(\hat \beta _1)\), with equality only when \(k_i = c_i\) for all \(i\). That is, \(\hat \beta _1\) is the best linear unbiased estimator (BLUE) of \(\beta _1\).
Proof. \(E(\tilde \beta _1) = \sum k_i(\beta _0 + \beta _1 x_i) = \beta _0 \sum k_i + \beta _1 \sum k_i x_i\). For this to equal \(\beta _1\) whatever the values of \(\beta _0\) and \(\beta _1\), we need \[ \sum k_i = 0 \qquad \text {and}\qquad \sum k_i x_i = 1 . \] Write \(k_i = c_i + d_i\), where \(d_i = k_i - c_i\). Since the \(c_i\) satisfy the same two conditions (Lemma 2.4.1), \(\sum d_i = 0\) and \(\sum d_i x_i = 0\). Hence \[ \sum c_i d_i = \frac {1}{S_{xx}} \sum (x_i - \bar x) d_i = \frac {1}{S_{xx}} \Bigl ( \sum x_i d_i - \bar x \sum d_i \Bigr ) = 0 , \] and so \begin {align*} \operatorname {Var}(\tilde \beta _1) &= \sigma ^2 \sum k_i^2 = \sigma ^2 \Bigl ( \sum c_i^2 + 2 \sum c_i d_i + \sum d_i^2 \Bigr ) \\ &= \operatorname {Var}(\hat \beta _1) + \sigma ^2 \sum d_i^2 . \end {align*}
The last term is at least zero, and it is zero only when every \(d_i = 0\), that is, when \(k_i = c_i\). □
The same is true of \(\hat \beta _0\); we prove the general result, for any number of regressors, in the section on multiple linear regression.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.