3.2 The model in matrix form

For a sample of \(n\) observations, the model (3.1) says \(y_i = \beta _0 + \beta _1 x_{i1} + \cdots + \beta _k x_{ik} + \varepsilon _i\), for \(i = 1, \dots , n\), where \(x_{ij}\) is the value of the \(j\)th regressor in the \(i\)th observation. Stack these \(n\) equations: \[ \underbrace {\begin {bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end {bmatrix}}_{\mathbf {y}} = \underbrace {\begin {bmatrix} 1 & x_{11} & \cdots & x_{1k} \\ 1 & x_{21} & \cdots & x_{2k} \\ \vdots & \vdots & & \vdots \\ 1 & x_{n1} & \cdots & x_{nk} \end {bmatrix}}_{\mathbf {X}} \underbrace {\begin {bmatrix} \beta _0 \\ \beta _1 \\ \vdots \\ \beta _k \end {bmatrix}}_{\boldsymbol \beta } + \underbrace {\begin {bmatrix} \varepsilon _1 \\ \varepsilon _2 \\ \vdots \\ \varepsilon _n \end {bmatrix}}_{\boldsymbol \varepsilon }, \] that is, \[ \boxed {\;\mathbf {y} = \mathbf {X}\boldsymbol \beta + \boldsymbol \varepsilon \;} \] The \(n \times p\) matrix \(\mathbf {X}\) is the model matrix (or design matrix). Its first column of ones carries the intercept, and each further column holds the values of one regressor.

The assumptions A1–A4 of simple regression become:

A1.

\(E(\boldsymbol \varepsilon ) = \mathbf {0}\);

A2–3.

\(\operatorname {Var}(\boldsymbol \varepsilon ) = \sigma ^2 \mathbf {I}\): the diagonal says every error has variance \(\sigma ^2\), the zeros off the diagonal say the errors are uncorrelated;

A4.

\(\boldsymbol \varepsilon \sim N_n(\mathbf {0}, \sigma ^2\mathbf {I})\): the errors are independent normal variables.

We also assume that \(\mathbf {X}\) has full column rank \(p\): no column is a linear combination of the others. This is the analogue of “the \(x_i\) are not all equal” in simple regression. By Lemma 3.1.3, A1–A3 give \(E(\mathbf {y}) = \mathbf {X}\boldsymbol \beta \) and \(\operatorname {Var}(\mathbf {y}) = \sigma ^2\mathbf {I}\).