Definition 3.1.1. Let \(Y\) be an \(N\times 1\) observable vector of random variables, \(X\) be an \(N\times K\) matrix of constants \((N>K)\), \(B\) be a vector of \(K\) parameters and \(\varepsilon \) be \(N\times 1\) vector of random variables (unobservable) errors. let these be related by the equation \begin {equation} \label {eq:glm}Y=XB + \varepsilon . \end {equation} \(E(\varepsilon )=0\) and \(Cov(\varepsilon )=\Sigma \)
These specifications define a general linear model.

The matrix is known as the design matrix. There are three special cases of the general linear model which depend on

i).

The distribution of the random vector \(\varepsilon \).

ii).

The structure of the covariance matrix \(\Sigma \) of \(\varepsilon \).

iii).

The rank and structure of the design matrix \(X\).

i).

Simple linear regression \[Y_i=\beta _0 +\beta _1X_i +e_i,\quad i=1,2,\ldots ,n\quad e_i\thicksim N(0,\sigma ^2)\] \[Y= \begin {pmatrix} Y_1 & Y_2 & \cdots & Y_n\\ \end {pmatrix}^t, \quad B= \begin {pmatrix} \beta _0 & \beta _1\\ \end {pmatrix}^t, \quad \varepsilon = \begin {pmatrix} e_1 & e_2 & \cdots & e_n\\ \end {pmatrix}^t \] \[X= \begin {pmatrix} 1 & X_1\\ 1 & X_2\\ \vdots & \vdots \\ 1 & X_n\\ \end {pmatrix} \] \[Y=XB +\varepsilon \qquad rank\underbrace {B}_{n\times k}\leq \min (n,k)\] \[\Sigma = \sigma ^2 I\] \(rank(X)=2\) unless the \(X_{i'}\)s are all equal.

ii).

Linear regression with two predictors. \[Y_i=\beta _0 + \beta _1 X_{i1} + \beta _2 X_{i2} + e_i\quad i=1,2,\ldots ,n\quad e_i\thicksim ^{iid} N(0,\sigma ^2)\] \[Y=XB+\varepsilon \] \(Y\) and \(\varepsilon \) are the same as in \((\)i.\()\)

\begin {align*} B & = \begin {pmatrix} \beta _0 & \beta _1 & \beta _2 \end {pmatrix}^t\quad K=3\\ \underbrace {X}_{n\times k} & = \begin {pmatrix} 1 & X_{11} & X_{12}\\ 1 & X_{21} & X_{22}\\ \vdots & \vdots & \vdots \\ \vdots & \vdots & \vdots \\ 1 & X_{n1} & X_{n2}\\ \end {pmatrix} \qquad rank(X)=3,\qquad \Sigma = \sigma ^2I\\ \end {align*}

B: ANOVA Model

i).

One-way classification, in its two equivalent parameterisations.

\(Y_{ij}=\mu _j + e_{ij}\)	\(Y_{ij}=\mu +\tau _j + e_{ij}\)
\(i=1,2,\ldots ,n\)	\(i=1,2,\ldots ,n\)
\(j=1,2,\ldots ,k\)	\(j=1,2,\ldots ,k\)
\(e_{ij}\thicksim ^{iid}N(0,\sigma ^2)\)	\( e_{ij}\thicksim N(0,\sigma ^2)\quad \sum \limits ^k_{j=1}\tau _j=0\)
\(\underbrace {Y}_{nk\times 1}=\underbrace {X}_{nk\times k}\underbrace {B}_{k\times 1}+\underbrace {\varepsilon }_{nk\times 1}\)	\(\underbrace {Y}_{nk\times 1}=\underbrace {X^*}_{nk\times (k+1)}\underbrace {B^*}_{(k+1)\times 1}+\underbrace {\varepsilon }_{nk\times 1}\)
\(k=3,\quad n=\) free
Let
\(Y^*_1= \begin {pmatrix} y_{11} & y_{21} & \cdots & y_{n1}\\ \end {pmatrix}^t\)	\(Y^*_1\), \(Y^*_2\) and \(Y^*_3\) same as on the left.
\(Y_2^*=\begin {pmatrix} y_{12} & y_{22} & \cdots & y_{n2}\\ \end {pmatrix}^t\)
\(Y_3^*= \begin {pmatrix} y_{13} & y_{23} & \cdots & y_{n3}\\ \end {pmatrix}^t \)
\(\varepsilon ^*_j= \begin {pmatrix} e_{1j} & e_{2j} & \cdots & e_{nj} \end {pmatrix}^t\quad j=1,2,3.\)	\(\varepsilon ^*_1\), \(\varepsilon ^*_2\) and \(\varepsilon ^*_3\) same as on the left
\(Y= \begin {pmatrix} Y^*_1 & Y^*_2 & Y_3^*\\ \end {pmatrix}^t= \begin {pmatrix} Y^*_1\\Y^*_2\\Y^*_3\\ \end {pmatrix}\)	\(Y\) vector same as on the left
\(\varepsilon = \begin {pmatrix} \varepsilon ^*_1 & \varepsilon ^*_2 & \varepsilon ^*_3\\ \end {pmatrix}^t\)	\(\varepsilon \) vector same as on the left.
\(X= \begin {pmatrix} 1_n & 0 & 0\\ 0 & 1_n & 0\\ 0 & 0 & 1_n\\ \end {pmatrix}\)	\(X^*= \begin {pmatrix} 1_n & 1_n & 0 & 0\\ 1_n & 0 & 1_n & 0\\ 1_n & 0 & 0 & 1_n\\ \end {pmatrix}\)
\(\underbrace {0}_{n\times 1}= \begin {pmatrix} 0 & 0 & \cdots & 0\\ \end {pmatrix}^t\)
\(\underbrace {1_n}_{n\times 1}= \begin {pmatrix} 1 & 1 & \cdots & 1\\ \end {pmatrix}^t\)
\(B= \begin {pmatrix} \mu _1\\ \mu _2\\ \mu _3\\ \end {pmatrix}_{3\times 1}\)	\( B= \begin {pmatrix} \mu \\ \tau _1\\ \tau _2\\ \tau _3\\ \end {pmatrix}_{4\times 1}\)

\(rank(X)=3\)	\(rank(X^*)=4\)
\(X^tX = \begin {pmatrix} 1^t_n & 0 &0\\ 0 & 1^t_n & 0\\ 0 & 0 & 1^t_n\\ \end {pmatrix} \begin {pmatrix} 1_n & 0 & 0\\ 0 & 1_n & 0\\ 0 & 0 & 1_n\\ \end {pmatrix} \)	\(X^{^*t}X^*= \begin {pmatrix} 1^t_n & 1^t_n & 1^t_n\\ 1^t_n & 0 & 0\\ 0 & 1^t_n & 0\\ 0 & 0 & 1^t_n\\ \end {pmatrix} \begin {pmatrix} 1_n & 1_n & 0 & 0\\ 1_n & 0 & 1_n & 0\\ 1_n & 0 & 0 & 1_n\\ \end {pmatrix} \)
\(= \begin {pmatrix} n & 0 & 0\\ 0 & n & 0\\ 0 & 0 & n\\ \end {pmatrix} \)	\(= \begin {pmatrix} 3n & n & n & n\\ n & n & 0 & 0\\ n & 0 & n & 0\\ n & 0 & 0 & n\\ \end {pmatrix} \)
\(=n \begin {pmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1\\ \end {pmatrix} \)	\(=n \begin {pmatrix} 3 & 1 & 1 & 1\\ 1 & 1 & 0 & 0\\ 1 & 0 & 1 & 0\\ 1 & 0 & 0 & 1\\ \end {pmatrix} \)
\(\displaystyle {\Sigma = \sigma ^2 I}\)	\(\displaystyle {\Sigma =\sigma ^2I}\)

ii).

\(Y_{ij} = \mu + \beta _i +\tau _j+e_{ij},\quad e_{ij}\thicksim ^{iid} N(0,\sigma ^2)\) \[i=1,2,\ldots , b\,, \quad j=1,2,\ldots ,k\] \[Y=XB+\varepsilon \] \(Y\) and \(\varepsilon \) are the same as in (i) \(Y_{ij}=\mu _j+e_{ij}\). \[k=3\quad b=4\] \[B_{(b+4)\times 1}= \begin {pmatrix} \mu \\ \beta _1\\ \vdots \vdots \\ \beta _b\\ \tau _1\\ \tau _2\\ \tau _3\\ \end {pmatrix} \] \[X_{3b\times (b+4)}= \begin {pmatrix} \mu & \beta _1 & \beta _2 & \beta _3 & \beta _4 & \tau _1 & \tau _2 & \tau _3\\ 1 & 1 & 0 & 0 & 0 & 1 & 0 & 0\\ 1 & 0 & 1 & 0 & 0 & 1 & 0 & 0\\ 1 & 0 & 0 & 1 & 0 & 1 & 0 & 0\\ 1 & 0 & 0 & 0 & 1 & 1 & 0 & 0\\ \end {pmatrix} \] \begin {align*} Y_{11} & = \mu + \beta _1 + \tau _1 + e_{11}\\ Y_{21} & = \mu + \beta _2 + \tau _1 + e_{21}\\ Y_{31} & = \mu + \beta _3 + \tau _1 + e_{31}\\ Y_{41} & = \mu + \beta _4 + \tau _1 + e_{41}\\ Y_{12} & = \mu + \beta _1 + \tau _2 + e_{12} \end {align*}

\begin {align*} X_{12\times 8} & = \begin {pmatrix} \mu & \beta _1 & \beta _2 & \beta _3 & \beta _4 & \tau _1 & \tau _2 & \tau _3\\ 1 & 1 & 0 & 0 & 0 & 1 & 0 & 0 \\ 1 & 0 & 1 & 0 & 0 & 1 & 0 & 0\\ 1 & 0 & 0 & 1 & 0 & 1 & 0 & 0\\ 1 & 0 & 0 & 0 & 1 & 1 & 0 & 0\\ - & - & - & - & - & - & - & -\\ 1 & 1 & 0 & 0 & 0 & 0 & 1 & 0\\ 1 & 0 & 1 & 0 & 0 & 0 & 1 & 0\\ 1 & 0 & 0 & 1 & 0 & 0 & 1 & 0\\ 1 & 0 & 0 & 0 & 1 & 0 & 1 & 0\\ - & - & - & - & - & - & - & -\\ 1 & 1 & 0 & 0 & 0 & 0 & 0 & 1\\ 1 & 0 & 1 & 0 & 0 & 0 & 0 & 1\\ 1 & 0 & 0 & 1 & 0 & 0 & 0 & 1\\ 1 & 0 & 0 & 0 & 1 & 0 & 0 & 1\\ \end {pmatrix} \quad 1_n\\ rank(X) & = 7\\ \end {align*}

Analysis of Covariance

\begin {align*} Y_{ij} & = \mu +\tau _j + \gamma X_{ij} + e_{ij}\\ Y_{ij} & = \mu _j +\gamma X_{ij} + e_{ij},\quad i=1,2,\ldots , n\quad j=1,2,\ldots , k\\ e_{ij} & \thicksim ^{iid} N(0,\sigma ^2) \end {align*}

\(Y_{ij}=i^{\text {th}}\) observation for the \(j^{\text {th}}\) treatment.

\(X_{ij}=i^{\text {th}}\) value of the covariance for experimental units on the \(j^{\text {th}}\) treatment. Measured before the experiment.

\(Y\) and \(\varepsilon \) are the same as in the model \(Y_{ij}=\mu +\beta _i + \tau _j + e_{ij}\)

\[B= \begin {pmatrix} \mu \\ \tau _1 \\ \tau _2\\ \vdots \\ \vdots \\ \tau _k \\ \gamma \\ \end {pmatrix}_{(k+2)\times 1} \quad B= \begin {pmatrix} \mu _1 \\ \mu _2\\ \vdots \\ \vdots \\ \mu _k\\ \gamma \\ \end {pmatrix}_{(k+1)\times 1} \] \begin {align*} X_{3n\times 5} & = \begin {pmatrix} \mu & \tau _1 & \tau _2 & \tau _3 & \gamma \\ 1 & 1 & 0 & 0 & X_{11}\\ 1 & 1 & 0 & 0 & X_{21}\\ \vdots & \vdots & \vdots & \vdots &\vdots \\ 1 & 1 & 0 & 0 & X_{n1}\\ - & - & - & - & -\\ 1 & 0 & 1 & 0 & X_{12}\\ 1 & 0 & 1 & 0 & X_{23}\\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 1 & 0 & 1 & 0 & X_{n2}\\ - & - & - & - & -\\ 1 & 0 & 0 & 1 & X_{13}\\ 1 & 0 & 0 & 1 & X_{23}\\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 1 & 0 & 0 & 1 & X_{n3}\\ \end {pmatrix}\\ \end {align*}