3.1 Best Linear Unbiased Estimators

An estimator is a best linear unbiased estimator (BLUE) if it is Linear and unbiased estimator with smallest variance.

Assume that \(Y = (Y_1, \, \cdots \, , \, Y_n)^t\) has expectation \(E(Y) = XB\) where \(X\) is some \(n\times k\) matrix having rank \(k\) and \(B = (\beta _1, \, \beta _2, \, \cdots \, , \, \beta _k)^t\) is a vector of unknown parameters. Suppose the convariance of \(Y\) is \(\sigma ^2 B\) where \(B\) is a non-singular matrix and \(\sigma ^2\) is scalar. We want to find the BLUE of a linear combination of \(B\), say \(\theta = a^t B\) where \(a\) is a \(k\)-dimensional vector.

Theorem 3.1.1 (Gauss–Markov Theorem). Let \(E(Y) = XB\) and \(\var (Y) = \sigma ^2 B\), then the best linear unbiased estimator (BLUE) of \(\theta = a^t B\) is \[\hat {\theta } = a^t \left (X^t B^{-1}X\right )^{-1}X^tB^{-1}Y.\]

Example 3.1.2. Suppose \(Y_1, \, \cdots \, , \, Y_n\) are independent unbiased estimators of \(\theta \) with unknown variance \(\var (Y_i) = \sigma ^2, \, \, i = 1, \, 2, \, \cdots \, , \, n\). Find the BLUE of \(\theta \).

Solution. \(Y = XB + \varepsilon \) \[Y_i = \theta + \varepsilon _i \, , \, \hspace {0.3cm} E(Y_i) = \theta \, , \, \hspace {0.3cm} E(\varepsilon _i) = 0\, , \, \hspace {0.2cm} \var (Y_i) = \var (\varepsilon _i) = \sigma ^2.\]

\[\begin {pmatrix} Y_1\\ \vdots \\ Y_n\\ \end {pmatrix} = \begin {pmatrix} 1\\ \vdots \\ 1\\ \end {pmatrix}\, (\theta ) + \begin {pmatrix} \varepsilon _1\\ \vdots \\ \varepsilon _n\\ \end {pmatrix}\]

\[\var (Y) = \sigma ^2\, I = \begin {pmatrix} \sigma ^2 & \cdots & 0\\ \vdots & & \vdots \\ 0 & \cdots & \sigma ^2\\ \end {pmatrix} = \sigma ^2\, \begin {pmatrix} 1 & \cdots & 0\\ \vdots & \vdots & \vdots \\ 0 & \cdots & 1\\ \end {pmatrix}.\]

\[\hat {\theta } = a^t\left (X^tB^{-1}X\right )^{-1}X^tB^{-1}Y\]

\[X^tB^{-1} = (1\,\, \cdots \, \, 1)\frac {1}{\sigma ^2}I = \frac {1}{\sigma ^2}\, (1\,\, \cdots \,\, 1).\]

\[X^tB^{-1}X = \frac {1}{\sigma ^2}\, (1 \, \cdots \, \, 1)\begin {pmatrix} 1\\ \vdots \\ 1\\ \end {pmatrix} = \frac {n}{\sigma ^2}.\]

\[\left (X^tB^{-1}X\right )^{-1} = \frac {\sigma ^2}{n}.\]

\[\left (X^tB^{-1}X\right )^{-1} = \frac {\sigma ^2}{n}\cdot \frac {1}{\sigma ^2}\, (1\, \cdots \, 1) = \frac {1}{n}\,(1\, \cdots \, 1).\] \begin {align*} \left (X^tB^{-1}X\right )^{-1}X^tB^{-1}Y & = \frac {1}{n}(1\, \, \cdots \, \, 1)\begin {pmatrix} Y_1\\ \vdots \\ Y_n\\ \end {pmatrix} = \frac {1}{n}\sum ^n_{i = 1} Y_i = \overline {Y}. \end {align*} □

Problem 3.1.1. Suppose \(Y_1, \, \cdots \, , \, Y_n\) are independent unbiased estimators of \(\theta \) with known variances \(\var (Y_i) = \sigma _i^2, \, i = 1, \, 2, \, \cdots \, , \, n\). Find the BLUE of \(\theta \).

Show solution

Solution. A linear estimator \(\sum _i a_iY_i\) is unbiased for \(\theta \) if and only if \(\sum _i a_i = 1\), since \(E\left (\sum _i a_iY_i\right )=\theta \sum _i a_i\). Its variance is \(\sum _i a_i^{2}\sigma _i^{2}\) by independence, so the problem is to minimise \(\sum _i a_i^{2}\sigma _i^{2}\) subject to \(\sum _i a_i=1\).

With a Lagrange multiplier \(\lambda \), differentiating \(\sum _i a_i^{2}\sigma _i^{2} - \lambda \left (\sum _i a_i - 1\right )\) gives \(2a_i\sigma _i^{2}=\lambda \), so \(a_i \propto 1/\sigma _i^{2}\). Normalising to make the weights sum to one, \[\widehat {\theta } = \frac {\sum _{i=1}^{n} Y_i/\sigma _i^{2}} {\sum _{i=1}^{n} 1/\sigma _i^{2}}, \qquad \var \left (\widehat {\theta }\right ) = \frac {1}{\sum _{i=1}^{n}1/\sigma _i^{2}} .\]

Note. Each estimator is weighted by its precision, the reciprocal of its variance, and the precision of the combination is the sum of the individual precisions. So adding even a poor estimator never hurts — it can only increase the total precision — which is the formal basis of meta-analysis and of inverse-variance weighting generally.

Problem 3.1.2. Suppose \(Y_{ij}, \, i = 1, \, 2, \ j = 1, \, 2, \, \cdots \, , n\) are independent random variables with \(E(Y_{ij}) = \mu + \alpha _i\) and \(\var (Y_{ij}) = \sigma ^2\) where \(\alpha _1 + \alpha _2 = 0\). Find the BLUE of \(\alpha _1\).

Show solution

Solution. The constraint \(\alpha _1+\alpha _2=0\) means \(\alpha _2=-\alpha _1\), so \[E\left (\overline {Y}_{1\cdot }\right ) = \mu +\alpha _1,\qquad E\left (\overline {Y}_{2\cdot }\right ) = \mu -\alpha _1 ,\] where \(\overline {Y}_{i\cdot }=n^{-1}\sum _{j=1}^{n}Y_{ij}\). Subtracting removes \(\mu \): \[E\left (\overline {Y}_{1\cdot }-\overline {Y}_{2\cdot }\right ) = 2\alpha _1 ,\] so \[\widehat {\alpha _1} = \frac {\overline {Y}_{1\cdot }-\overline {Y}_{2\cdot }}{2}\] is linear and unbiased. Its variance is \[\var \left (\widehat {\alpha _1}\right ) = \frac {1}{4}\left (\frac {\sigma ^{2}}{n}+\frac {\sigma ^{2}}{n}\right ) = \frac {\sigma ^{2}}{2n} .\] That it is best follows from the Gauss–Markov theorem: the model is linear with uncorrelated errors of constant variance, and \(\widehat {\alpha _1}\) is the least squares estimator of the estimable function \(\alpha _1\) under the constraint.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.