7.1 The sampling distribution of \(b\)
Theorem 7.1.1. In the model \(Y = X\beta + U\) with \(\rank X = k\):
- (i)
- \(b = \beta + (X'X)^{-1}X'U\);
- (ii)
- under (A1), \(E(b \mid X) = \beta \);
- (iii)
- under (A1)–(A3), \(\Var (b \mid X) = \sigma ^2(X'X)^{-1}\);
- (iv)
- under (A1)–(A5), \(Y \sim N(X\beta , \sigma ^2I_n)\) and \(b \sim N\bigl (\beta , \sigma ^2(X'X)^{-1}\bigr )\), conditionally on \(X\).
Proof. (i) \(b = (X'X)^{-1}X'(X\beta + U) = \beta + (X'X)^{-1}X'U\). (ii) \(E(b \mid X) = \beta + (X'X)^{-1}X'E(U \mid X) = \beta \). (iii) By Theorem 5.2.1(i) with \(A = (X'X)^{-1}X'\), \[ \Var (b \mid X) = (X'X)^{-1}X'(\sigma ^2I)X(X'X)^{-1} = \sigma ^2(X'X)^{-1} . \] (iv) \(Y = X\beta + U\) is a linear function of the normal vector \(U\), and \(b = \beta + AU\) is a linear function of it with \(A\) of full row rank \(k\). Apply Theorem 5.3.2(ii). □
The \(j\)th diagonal element of \(\sigma ^2(X'X)^{-1}\) is \(\Var (b_j)\), and the off-diagonal elements are the covariances between the estimates. Those covariances matter whenever a hypothesis involves more than one coefficient. They are usually large when the corresponding regressors are correlated (Problem 7.8.5). Replacing \(\sigma ^2\) by \(s^2 = e'e/(n - k)\), unbiased by Theorem 5.5.1, gives the estimated covariance matrix \(s^2(X'X)^{-1}\), whose diagonal square roots are the standard errors printed by every regression program. The diagonal element of \((X'X)^{-1}\) for regressor \(j\) can be written \(1/\bigl (\sum x_j^2(1 - R_j^2)\bigr )\), where \(R_j^2\) is the \(R^2\) of regressing \(X_j\) on the other regressors. That shows directly how correlation among the regressors inflates the variance; it is proved by the Frisch–Waugh–Lovell theorem in Problem 7.8.6.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.