3.1 Matrix results we shall need
Vectors are columns and are printed in bold: \(\mathbf {y}\), \(\boldsymbol \beta \). The transpose of \(\mathbf {A}\) is \(\mathbf {A}^\top \), and \((\mathbf {AB})^\top = \mathbf {B}^\top \mathbf {A}^\top \). The trace of a square matrix is the sum of its diagonal entries, \(\operatorname {tr}(\mathbf {A}) = \sum _i a_{ii}\).
Lemma 3.1.1 (Trace). If \(\mathbf {A}\) is \(m \times n\) and \(\mathbf {B}\) is \(n \times m\), then \(\operatorname {tr}(\mathbf {AB}) = \operatorname {tr}(\mathbf {BA})\).
Proof. \(\operatorname {tr}(\mathbf {AB}) = \sum _{i=1}^m (\mathbf {AB})_{ii} = \sum _{i=1}^m \sum _{j=1}^n a_{ij} b_{ji}\) and \(\operatorname {tr}(\mathbf {BA}) = \sum _{j=1}^n \sum _{i=1}^m b_{ji} a_{ij}\). These are the same sum taken in a different order. □
Lemma 3.1.2 (Differentiation). For a vector \(\boldsymbol \beta = (\beta _0, \dots , \beta _k)^\top \), let \(\partial f / \partial \boldsymbol \beta \) denote the vector of partial derivatives \(\partial f / \partial \beta _j\). Then, for a constant vector \(\mathbf {a}\) and a constant symmetric matrix \(\mathbf {A}\), \[ \frac {\partial (\mathbf {a}^\top \boldsymbol \beta )}{\partial \boldsymbol \beta } = \mathbf {a}, \qquad \frac {\partial (\boldsymbol \beta ^\top \mathbf {A} \boldsymbol \beta )}{\partial \boldsymbol \beta } = 2\mathbf {A}\boldsymbol \beta . \]
Proof. \(\mathbf {a}^\top \boldsymbol \beta = \sum _j a_j \beta _j\), whose derivative with respect to \(\beta _m\) is \(a_m\). Next, \[ \boldsymbol \beta ^\top \mathbf {A} \boldsymbol \beta = \sum _i \sum _j a_{ij} \beta _i \beta _j . \] Differentiating with respect to \(\beta _m\), the terms with \(i = m\) give \(\sum _j a_{mj}\beta _j\) and the terms with \(j = m\) give \(\sum _i a_{im}\beta _i\). Since \(a_{im} = a_{mi}\), both equal the \(m\)th entry of \(\mathbf {A}\boldsymbol \beta \), so the derivative is \(2(\mathbf {A}\boldsymbol \beta )_m\). □
For a random vector \(\mathbf {y} = (y_1, \dots , y_n)^\top \), the mean vector \(E(\mathbf {y})\) has entries \(E(y_i)\), and the covariance matrix \[ \operatorname {Var}(\mathbf {y}) = E\left [ (\mathbf {y} - E\mathbf {y})(\mathbf {y} - E\mathbf {y})^\top \right ] \] has \((i,j)\) entry \(\operatorname {Cov}(y_i, y_j)\), and so has the variances \(\operatorname {Var}(y_i)\) down its diagonal.
Lemma 3.1.3 (Linear functions of a random vector). If \(\mathbf {A}\) is a constant matrix and \(\mathbf {b}\) a constant vector, then \[ E(\mathbf {A}\mathbf {y} + \mathbf {b}) = \mathbf {A}E(\mathbf {y}) + \mathbf {b}, \qquad \operatorname {Var}(\mathbf {A}\mathbf {y} + \mathbf {b}) = \mathbf {A} \operatorname {Var}(\mathbf {y}) \mathbf {A}^\top . \]
Proof. The \(i\)th entry of \(\mathbf {Ay} + \mathbf {b}\) is \(\sum _j a_{ij} y_j + b_i\), whose expectation is \(\sum _j a_{ij} E(y_j) + b_i\), the \(i\)th entry of \(\mathbf {A}E(\mathbf {y}) + \mathbf {b}\). Writing \(\boldsymbol \mu = E(\mathbf {y})\), we have \(\mathbf {Ay} + \mathbf {b} - E(\mathbf {Ay} + \mathbf {b}) = \mathbf {A}(\mathbf {y} - \boldsymbol \mu )\), so \[ \operatorname {Var}(\mathbf {Ay} + \mathbf {b}) = E\left [ \mathbf {A}(\mathbf {y} - \boldsymbol \mu )(\mathbf {y} - \boldsymbol \mu )^\top \mathbf {A}^\top \right ] = \mathbf {A} \operatorname {Var}(\mathbf {y}) \mathbf {A}^\top , \] taking the constant matrices outside the expectation. □
Lemma 3.1.4 (Expected quadratic form). If \(E(\mathbf {y}) = \boldsymbol \mu \), \(\operatorname {Var}(\mathbf {y}) = \mathbf {V}\) and \(\mathbf {A}\) is a constant matrix, then \[ E(\mathbf {y}^\top \mathbf {A} \mathbf {y}) = \operatorname {tr}(\mathbf {AV}) + \boldsymbol \mu ^\top \mathbf {A} \boldsymbol \mu . \]
Proof. Write \(\mathbf {y} = \boldsymbol \mu + \mathbf {z}\), where \(E(\mathbf {z}) = \mathbf {0}\) and \(E(\mathbf {z}\mathbf {z}^\top ) = \mathbf {V}\). Then \[ \mathbf {y}^\top \mathbf {A} \mathbf {y} = \boldsymbol \mu ^\top \mathbf {A} \boldsymbol \mu + \boldsymbol \mu ^\top \mathbf {A} \mathbf {z} + \mathbf {z}^\top \mathbf {A} \boldsymbol \mu + \mathbf {z}^\top \mathbf {A} \mathbf {z} . \] The two middle terms have expectation \(0\). The last is a \(1 \times 1\) matrix, so equals its own trace, and by Lemma 3.1.1 \(\mathbf {z}^\top \mathbf {A} \mathbf {z} = \operatorname {tr}(\mathbf {z}^\top \mathbf {A} \mathbf {z}) = \operatorname {tr}(\mathbf {A} \mathbf {z}\mathbf {z}^\top )\). Since the trace is a sum of entries, it can be taken through the expectation: \(E\operatorname {tr}(\mathbf {A}\mathbf {z}\mathbf {z}^\top ) = \operatorname {tr}(\mathbf {A}\,E(\mathbf {z}\mathbf {z}^\top )) = \operatorname {tr}(\mathbf {AV})\). □
Finally, from linear algebra: a square matrix \(\mathbf {A}\) has an inverse exactly when \(\det \mathbf {A} \neq 0\), and then \[ \mathbf {A}^{-1} = \frac {1}{\det \mathbf {A}} \operatorname {adj}(\mathbf {A}), \] where the adjugate \(\operatorname {adj}(\mathbf {A})\) is the transpose of the matrix of cofactors. The cofactor of \(a_{ij}\) is \((-1)^{i+j}\) times the determinant of the matrix left when row \(i\) and column \(j\) are deleted. For a symmetric matrix, such as the ones met in regression, the matrix of cofactors is itself symmetric and no transposing is needed.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.