3.4 Estimability and the Generalised Inverse
Everything so far has assumed that \(X\) has full column rank, so that \(\left (X^tX\right )^{-1}\) exists and the normal equations have one solution. The analysis of variance models of Section 1 do not satisfy that assumption. In the one-way layout \[Y_{ij}=\mu +\tau _j+\varepsilon _{ij},\] there are \(k+1\) parameters \(\mu ,\tau _1,\dots ,\tau _k\) but the design matrix has rank \(k\): its first column is the sum of the others. The normal equations then have infinitely many solutions, and no amount of data will separate \(\mu \) from the \(\tau _j\) — adding a constant to \(\mu \) and subtracting it from every \(\tau _j\) leaves every fitted value unchanged.
This is not a defect to be repaired. It is a statement about which questions the experiment can answer. Some functions of the parameters are determined by the data and some are not, and this subsection says exactly which.
Definition 3.4.1 (Generalised inverse). A matrix \(A^{-}\) is a generalised inverse of \(A\) if \[A A^{-} A = A.\] When \(A\) is non-singular the only such matrix is \(A^{-1}\); otherwise \(A^{-}\) exists but is not unique.
Result 3.4.2. The normal equations \(X^tX\underline {b}=X^t\underline {Y}\) are always consistent, and every solution has the form \[\widehat {\underline {b}}=\left (X^tX\right )^{-}X^t\underline {Y}\] for some generalised inverse \(\left (X^tX\right )^{-}\). Different generalised inverses give different solutions.
Note. Consistency is the important half: however deficient \(X\) may be, a least squares solution always exists. What fails without full rank is uniqueness, and uniqueness is not needed for the parts of the answer that matter, as the next definition makes precise. Note also that the fitted values are unique regardless, since \(X\widehat {\underline {b}}=H\underline {Y}\) with \(H=X\left (X^tX\right )^{-}X^t\) the projection onto the column space of \(X\) — and a projection does not depend on which generalised inverse produced it.
Definition 3.4.3 (Estimable function). A linear function \(\underline {\lambda }^tB\) of the parameters is estimable if there exists a vector \(\underline {a}\) with \[E\left (\underline {a}^t\underline {Y}\right )=\underline {\lambda }^tB \qquad \text {for every }B,\] that is, if some linear function of the observations is an unbiased estimator of it.
Theorem 3.4.4 (Criterion for estimability). \(\underline {\lambda }^tB\) is estimable if and only if \(\underline {\lambda }^t\) lies in the row space of \(X\); equivalently, if and only if \[\underline {\lambda }^t\left (X^tX\right )^{-}\left (X^tX\right ) =\underline {\lambda }^t .\]
Proof. If \(\underline {\lambda }^tB\) is estimable then \(\underline {a}^tXB=\underline {\lambda }^tB\) for all \(B\), so \(\underline {\lambda }^t=\underline {a}^tX\) and \(\underline {\lambda }^t\) is a combination of the rows of \(X\). Conversely if \(\underline {\lambda }^t=\underline {a}^tX\) then \(E\left (\underline {a}^t\underline {Y}\right )=\underline {a}^tXB =\underline {\lambda }^tB\). The second form follows because \(\left (X^tX\right )^{-}\left (X^tX\right )\) acts as the identity exactly on the row space of \(X\). □
Theorem 3.4.5 (Gauss–Markov, rank-deficient form). If \(\underline {\lambda }^tB\) is estimable then \(\underline {\lambda }^t\widehat {\underline {b}}\) is the same for every solution \(\widehat {\underline {b}}\) of the normal equations, is unbiased for \(\underline {\lambda }^tB\), and has smaller variance than any other linear unbiased estimator of it. Its variance is \[\text {Var}\left (\underline {\lambda }^t\widehat {\underline {b}}\right ) =\sigma ^2\,\underline {\lambda }^t\left (X^tX\right )^{-}\underline {\lambda }.\]
Note. The invariance is the point. A computer package solving a rank-deficient analysis of variance must pick some generalised inverse, and different packages pick different ones — so the printed values of \(\widehat {\mu }\) and \(\widehat {\tau }_j\) may differ between two programs run on the same data. The estimable functions do not differ. If two analyses disagree about a quantity, that quantity was not estimable, and the disagreement is the software telling you so.
Example 3.4.6. In the one-way layout with \(k\) treatments, write \(B=\left (\mu ,\tau _1,\dots ,\tau _k\right )^t\). The rows of \(X\) are of the form \(\left (1,0,\dots ,1,\dots ,0\right )\) with the \(1\) in position \(j+1\), so the row space consists of all \(\underline {\lambda }^t=\left (c,c_1,\dots ,c_k\right )\) with \(c=\sum _j c_j\). Hence
- \(\mu +\tau _j\) is estimable, with \(c=1\), \(c_j=1\): it is the mean of treatment \(j\), and \(\overline {Y}_{\cdot j}\) estimates it.
- \(\tau _i-\tau _j\) is estimable, with \(c=0\): any difference of treatment effects is estimable, and \(\overline {Y}_{\cdot i}-\overline {Y}_{\cdot j}\) estimates it.
- \(\mu \) alone is not estimable: it would need \(c=1\) with every \(c_j=0\), and \(0\neq 1\).
- \(\tau _j\) alone is not estimable, by the same arithmetic.
Note. Example 3.4.6 explains why the whole of Section 1 is written in terms of differences between treatments and never in terms of a single \(\tau _j\). It also puts the familiar side condition \(\sum _j\tau _j=0\) in its place: it is a convention that selects one solution out of the infinitely many, not information extracted from the data. Imposing it makes \(\mu \) and \(\tau _j\) individually computable, but their values then describe the convention as much as the experiment. Any quantity whose value changes when the side condition is changed was never estimable.
Definition 3.4.7 (Contrast). A contrast is an estimable function \(\sum _j c_j\tau _j\) with \(\sum _j c_j=0\). Two contrasts \(\underline {c}\) and \(\underline {d}\) are orthogonal if \(\sum _j c_jd_j/n_j=0\); in the balanced case this is \(\sum _j c_jd_j=0\).
Note. By Example 3.4.6 every contrast is estimable, and the contrasts are precisely the estimable functions of the \(\tau _j\) alone. A set of \(k-1\) mutually orthogonal contrasts partitions \(SSTr\) into \(k-1\) single-degree-of-freedom sums of squares that add to it exactly — so a significant overall \(F\) can be followed by an examination of which contrast carries the effect, with no multiplicity penalty if the contrasts were chosen before the data were seen.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.