3.5 Properties of the least-squares estimator
Theorem 3.5.1. Under A1–A3, \(\hat {\boldsymbol \beta }\) is unbiased, with covariance matrix \[ E(\hat {\boldsymbol \beta }) = \boldsymbol \beta , \qquad \operatorname {Var}(\hat {\boldsymbol \beta }) = \sigma ^2(\mathbf {X}^\top \mathbf {X})^{-1} = \sigma ^2\mathbf {C} . \] So \(\operatorname {Var}(\hat \beta _j) = \sigma ^2 C_{jj}\) and \(\operatorname {Cov}(\hat \beta _i, \hat \beta _j) = \sigma ^2 C_{ij}\), where the rows and columns of \(\mathbf {C}\) are numbered \(0, 1, \dots , k\) to match the coefficients.
Proof. \(\hat {\boldsymbol \beta } = \mathbf {C}\mathbf {X}^\top \mathbf {y}\) is a linear function of \(\mathbf {y}\). By Lemma 3.1.3, \(E(\hat {\boldsymbol \beta }) = \mathbf {C}\mathbf {X}^\top \mathbf {X}\boldsymbol \beta = \boldsymbol \beta \), and \[ \operatorname {Var}(\hat {\boldsymbol \beta }) = \mathbf {C}\mathbf {X}^\top (\sigma ^2\mathbf {I})\mathbf {X}\mathbf {C}^\top = \sigma ^2\mathbf {C}(\mathbf {X}^\top \mathbf {X})\mathbf {C} = \sigma ^2\mathbf {C} . \qedhere \] □
For simple regression, \(\sigma ^2\mathbf {C}\) is \(\sigma ^2/(nS_{xx})\) times the matrix in Example 3.3.2, and its entries are exactly the variances and covariance of Theorems 2.4.2 and 2.4.4 (Problem 3.13.3).
Theorem 3.5.2 (Gauss–Markov). Under A1–A3, let \(\mathbf {c}\) be any constant vector. Among all estimators of \(\mathbf {c}^\top \boldsymbol \beta \) that are linear in \(\mathbf {y}\) and unbiased, \(\mathbf {c}^\top \hat {\boldsymbol \beta }\) has the smallest variance. In particular each \(\hat \beta _j\) is the best linear unbiased estimator of \(\beta _j\).
Proof. Let \(\mathbf {a}^\top \mathbf {y}\) be linear and unbiased for \(\mathbf {c}^\top \boldsymbol \beta \): \(E(\mathbf {a}^\top \mathbf {y}) = \mathbf {a}^\top \mathbf {X}\boldsymbol \beta = \mathbf {c}^\top \boldsymbol \beta \) for every \(\boldsymbol \beta \), which forces \(\mathbf {X}^\top \mathbf {a} = \mathbf {c}\). The least-squares estimator is \(\mathbf {c}^\top \hat {\boldsymbol \beta } = \mathbf {a}_0^\top \mathbf {y}\) with \(\mathbf {a}_0 = \mathbf {X}\mathbf {C}\mathbf {c}\). Let \(\mathbf {d} = \mathbf {a} - \mathbf {a}_0\). Then \[ \mathbf {X}^\top \mathbf {d} = \mathbf {c} - \mathbf {X}^\top \mathbf {X}\mathbf {C}\mathbf {c} = \mathbf {c} - \mathbf {c} = \mathbf {0}, \qquad \text {so}\qquad \mathbf {a}_0^\top \mathbf {d} = \mathbf {c}^\top \mathbf {C}\mathbf {X}^\top \mathbf {d} = 0 . \] Since \(\operatorname {Var}(\mathbf {y}) = \sigma ^2\mathbf {I}\), Lemma 3.1.3 gives \(\operatorname {Var}(\mathbf {a}^\top \mathbf {y}) = \sigma ^2\mathbf {a}^\top \mathbf {a}\), and \[ \mathbf {a}^\top \mathbf {a} = (\mathbf {a}_0 + \mathbf {d})^\top (\mathbf {a}_0 + \mathbf {d}) = \mathbf {a}_0^\top \mathbf {a}_0 + \mathbf {d}^\top \mathbf {d} \ge \mathbf {a}_0^\top \mathbf {a}_0 . \] So \(\operatorname {Var}(\mathbf {a}^\top \mathbf {y}) \ge \operatorname {Var}(\mathbf {c}^\top \hat {\boldsymbol \beta })\), with equality only when \(\mathbf {d} = \mathbf {0}\). Taking \(\mathbf {c}\) with a \(1\) in place \(j\) and zeros elsewhere gives the statement about \(\hat \beta _j\). □
This also settles the claim made after Theorem 2.4.5: in simple regression \(\hat \beta _0\), as well as \(\hat \beta _1\), is best linear unbiased.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.