3.1 Best Linear Unbiased Estimators
An estimator is a best linear unbiased estimator (BLUE) if it is a linear function of the observations, is unbiased, and has the smallest variance among all linear unbiased estimators.
Suppose \(Y = (Y_1, \dots , Y_n)^{\mathsf T}\) has expectation \(E(Y) = X\beta \), where \(X\) is a known \(n\times k\) matrix of rank \(k\) and \(\beta = (\beta _1, \dots , \beta _k)^{\mathsf T}\) is a vector of unknown parameters, and covariance matrix \(\var (Y) = \sigma ^2V\), where \(V\) is a known non-singular matrix and \(\sigma ^2\) an unknown scalar. We want the BLUE of a linear combination \(\theta = a^{\mathsf T}\beta \), where \(a\) is a known \(k\)-vector.
Theorem 3.1.1 (Gauss–Markov). Under these assumptions the BLUE of \(\theta = a^{\mathsf T}\beta \) is \[\hat \theta = a^{\mathsf T}\hat \beta ,\qquad \hat \beta = \left (X^{\mathsf T}V^{-1}X\right )^{-1}X^{\mathsf T}V^{-1}Y,\] and its variance is \(\sigma ^2a^{\mathsf T}\left (X^{\mathsf T}V^{-1}X\right )^{-1}a\).
Proof. Write \(M = \left (X^{\mathsf T}V^{-1}X\right )^{-1}\), which exists because \(X\) has full rank and \(V\) is positive definite. The estimator \(\hat \theta = c_0^{\mathsf T}Y\) with \(c_0 = V^{-1}XMa\) is linear, and \(E(\hat \theta ) = a^{\mathsf T}MX^{\mathsf T}V^{-1}X\beta = a^{\mathsf T}\beta \), so it is unbiased. Any linear estimator \(c^{\mathsf T}Y\) has mean \(c^{\mathsf T}X\beta \), and it is unbiased for \(a^{\mathsf T}\beta \) for every \(\beta \) exactly when \(X^{\mathsf T}c = a\). Write \(c = c_0 + d\); since \(X^{\mathsf T}c_0 = X^{\mathsf T}V^{-1}XMa = a\), unbiasedness forces \(X^{\mathsf T}d = 0\). Then \begin {align*} \var (c^{\mathsf T}Y) &= \sigma ^2(c_0 + d)^{\mathsf T}V(c_0 + d)\\ &= \sigma ^2\left [c_0^{\mathsf T}Vc_0 + 2d^{\mathsf T}Vc_0 + d^{\mathsf T}Vd\right ]. \end {align*}
The cross term vanishes because \(Vc_0 = XMa\) and \(d^{\mathsf T}X = 0\). So \(\var (c^{\mathsf T}Y) = \var (\hat \theta ) + \sigma ^2d^{\mathsf T}Vd\ge \var (\hat \theta )\), with equality only when \(d = 0\). Finally \(\var (\hat \theta ) = \sigma ^2c_0^{\mathsf T}Vc_0 = \sigma ^2a^{\mathsf T}MX^{\mathsf T}V^{-1}XMa = \sigma ^2a^{\mathsf T}Ma\). □
When \(V = I\) the BLUE is the ordinary least squares estimator \(\hat \beta = (X^{\mathsf T}X)^{-1}X^{\mathsf T}Y\); for general \(V\) it is the generalised least squares (Aitken) estimator. The theorem needs no distributional assumption beyond the first two moments, but it compares only linear estimators. For normal errors, \(\hat \beta \) is also the UMVUE, so nothing is lost by the restriction; for heavy-tailed errors a nonlinear estimator can be much better.
Example 3.1.2. Let \(Y_1, \dots , Y_n\) be uncorrelated, each with mean \(\theta \) and the same unknown variance \(\sigma ^2\). Find the BLUE of \(\theta \).
Solution. In the notation of theorem 3.1.1, \(k = 1\), \(\beta = \theta \), \(a = 1\), \(X = \mathbf 1 = (1, \dots , 1)^{\mathsf T}\) and \(V = I\). Then \(X^{\mathsf T}V^{-1}X = \mathbf 1^{\mathsf T}\mathbf 1 = n\) and \(X^{\mathsf T}V^{-1}Y = \sum Y_i\), so \[\hat \theta = \frac 1n\sum _{i=1}^n Y_i = \bar Y,\qquad \var (\hat \theta ) = \frac {\sigma ^2}{n} .\]
Example 3.1.3. Let \(Y_i = \beta _0 + \beta _1x_i + \varepsilon _i\), \(i = 1, \dots , n\), where the \(x_i\) are known constants, not all equal, and the \(\varepsilon _i\) are uncorrelated with mean \(0\) and variance \(\sigma ^2\). Find the BLUE of the slope \(\beta _1\) and its variance.
Solution. Here \(V = I\) and \(X\) has rows \((1, x_i)\), so \[X^{\mathsf T}X = \begin {pmatrix} n & \sum x_i\\ \sum x_i & \sum x_i^2\end {pmatrix},\qquad X^{\mathsf T}Y = \begin {pmatrix}\sum Y_i\\ \sum x_iY_i\end {pmatrix}.\] The determinant is \(n\sum x_i^2 - \left (\sum x_i\right )^2 = nS_{xx}\), where \(S_{xx} = \sum (x_i - \bar x)^2>0\). Taking \(a = (0, 1)^{\mathsf T}\), \[\hat \beta _1 = \frac {n\sum x_iY_i - \sum x_i\sum Y_i}{nS_{xx}} = \frac {\sum (x_i - \bar x)Y_i}{S_{xx}},\qquad \var (\hat \beta _1) = \sigma ^2\left [(X^{\mathsf T}X)^{-1}\right ]_{22} = \frac {\sigma ^2}{S_{xx}} .\] This is the least squares slope. The variance is smallest when the \(x_i\) are spread out, which is why experiments to estimate a slope put the design points at the extremes of the range.
Problem 3.1.1. Suppose \(Y_1, \, \cdots \, , \, Y_n\) are independent unbiased estimators of \(\theta \) with known variances \(\var (Y_i) = \sigma _i^2, \, i = 1, \, 2, \, \cdots \, , \, n\). Find the BLUE of \(\theta \).
Show solution
Solution. A linear estimator \(\sum _i a_iY_i\) is unbiased for \(\theta \) if and only if \(\sum _i a_i = 1\), since \(E\left (\sum _i a_iY_i\right )=\theta \sum _i a_i\). Its variance is \(\sum _i a_i^{2}\sigma _i^{2}\) by independence, so the problem is to minimise \(\sum _i a_i^{2}\sigma _i^{2}\) subject to \(\sum _i a_i=1\).
With a Lagrange multiplier \(\lambda \), differentiating \(\sum _i a_i^{2}\sigma _i^{2} - \lambda \left (\sum _i a_i - 1\right )\) gives \(2a_i\sigma _i^{2}=\lambda \), so \(a_i \propto 1/\sigma _i^{2}\). Normalising to make the weights sum to one, \[\widehat {\theta } = \frac {\sum _{i=1}^{n} Y_i/\sigma _i^{2}} {\sum _{i=1}^{n} 1/\sigma _i^{2}}, \qquad \var \left (\widehat {\theta }\right ) = \frac {1}{\sum _{i=1}^{n}1/\sigma _i^{2}} .\]
Note. Each estimator is weighted by its precision, the reciprocal of its variance, and the precision of the combination is the sum of the individual precisions. So adding even a poor estimator never hurts — it can only increase the total precision — which is the formal basis of meta-analysis and of inverse-variance weighting generally.
Problem 3.1.2. Suppose \(Y_{ij}, \, i = 1, \, 2, \ j = 1, \, 2, \, \cdots \, , n\) are independent random variables with \(E(Y_{ij}) = \mu + \alpha _i\) and \(\var (Y_{ij}) = \sigma ^2\) where \(\alpha _1 + \alpha _2 = 0\). Find the BLUE of \(\alpha _1\).
Show solution
Solution. The constraint \(\alpha _1+\alpha _2=0\) means \(\alpha _2=-\alpha _1\), so \[E\left (\overline {Y}_{1\cdot }\right ) = \mu +\alpha _1,\qquad E\left (\overline {Y}_{2\cdot }\right ) = \mu -\alpha _1 ,\] where \(\overline {Y}_{i\cdot }=n^{-1}\sum _{j=1}^{n}Y_{ij}\). Subtracting removes \(\mu \): \[E\left (\overline {Y}_{1\cdot }-\overline {Y}_{2\cdot }\right ) = 2\alpha _1 ,\] so \[\widehat {\alpha _1} = \frac {\overline {Y}_{1\cdot }-\overline {Y}_{2\cdot }}{2}\] is linear and unbiased. Its variance is \[\var \left (\widehat {\alpha _1}\right ) = \frac {1}{4}\left (\frac {\sigma ^{2}}{n}+\frac {\sigma ^{2}}{n}\right ) = \frac {\sigma ^{2}}{2n} .\] That it is best follows from the Gauss–Markov theorem: the model is linear with uncorrelated errors of constant variance, and \(\widehat {\alpha _1}\) is the least squares estimator of the estimable function \(\alpha _1\) under the constraint.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.