5.1 Matrix results
Vectors are columns; a prime denotes transposition; \(I_n\) is the \(n \times n\) identity and \(\iota \) the \(n\)-vector of ones. All matrices are real.
Definition 5.1.1. The trace of a square matrix is the sum of its diagonal elements. A symmetric matrix \(A\) is positive definite (pd) if \(c'Ac > 0\) for every \(c \neq 0\), and positive semi-definite (psd) if \(c'Ac \geq 0\) for every \(c\). A square matrix is idempotent if \(AA = A\).
Lemma 5.1.2. \(\tr (A + B) = \tr A + \tr B\), \(\tr (cA) = c\tr A\), and \(\tr (AB) = \tr (BA)\) whenever both products are defined. In particular, for a vector \(x\), \(x'Ax = \tr (x'Ax) = \tr (Axx')\).
Proof. The first two are immediate. If \(A\) is \(m \times n\) and \(B\) is \(n \times m\), then \(\tr (AB) = \sum _i\sum _j a_{ij}b_{ji} = \sum _j\sum _i b_{ji}a_{ij} = \tr (BA)\). A scalar is its own trace, so \(x'Ax = \tr (x'(Ax)) = \tr ((Ax)x')\). □
The trace identity looks trivial. Its use is to move a random vector to the end of a product, where expectation can be applied to \(xx'\) alone, and it is behind every expected value of a quadratic form in this course.
Theorem 5.1.3 (Spectral decomposition). A real symmetric \(n \times n\) matrix \(A\) can be written \(A = Q\Lambda Q'\), where \(Q\) is orthogonal (\(Q'Q = QQ' = I\)) and \(\Lambda = \operatorname {diag}(\lambda _1, \dots , \lambda _n)\) holds the eigenvalues of \(A\), which are real. The columns of \(Q\) are corresponding orthonormal eigenvectors. The rank of \(A\) is the number of non-zero eigenvalues.
This is a standard theorem of linear algebra and we use it without proof. Three consequences follow, all used repeatedly.
Corollary 5.1.4. Let \(A\) be symmetric with spectral decomposition \(Q\Lambda Q'\).
- (i)
- \(A\) is pd if and only if every \(\lambda _i > 0\), and psd if and only if every \(\lambda _i \geq 0\).
- (ii)
- If \(A\) is pd it is non-singular, and \(A^{-1} = Q\Lambda ^{-1}Q'\) is pd.
- (iii)
- If \(A\) is pd, the matrix \(A^{1/2} = Q\Lambda ^{1/2}Q'\) is symmetric and pd, with \(A^{1/2}A^{1/2} = A\). Its inverse \(A^{-1/2} = Q\Lambda ^{-1/2}Q'\) satisfies \(A^{-1/2}AA^{-1/2} = I\). More generally, any non-singular \(P\) with \(P'P = A^{-1}\) satisfies \(PAP' = I\).
Proof. (i) With \(d = Q'c\), which is non-zero exactly when \(c\) is, \(c'Ac = d'\Lambda d = \sum _i\lambda _id_i^2\). This is positive for all \(d \neq 0\) if and only if every \(\lambda _i > 0\), and non-negative for all \(d\) if and only if every \(\lambda _i \geq 0\).
(ii) \(Q\Lambda ^{-1}Q'\,Q\Lambda Q' = Q\Lambda ^{-1}\Lambda Q' = I\), and the eigenvalues \(1/\lambda _i\) are positive.
(iii) \(A^{1/2}A^{1/2} = Q\Lambda ^{1/2}Q'Q\Lambda ^{1/2}Q' = Q\Lambda Q' = A\), and similarly for the inverse. If \(P'P = A^{-1}\) with \(P\) non-singular, then \(A = (P'P)^{-1} = P^{-1}(P')^{-1}\), so \(PAP' = PP^{-1}(P')^{-1}P' = I\). □
The last statement is the key to generalised least squares: a covariance matrix \(A\) that is not a multiple of the identity can be turned into the identity by a linear transformation \(P\), which may be the symmetric \(A^{-1/2}\) or any other factor of \(A^{-1}\), such as a Cholesky factor.
Theorem 5.1.5 (Idempotent matrices). Let \(A\) be symmetric and idempotent. Then
- (i)
- every eigenvalue of \(A\) is \(0\) or \(1\);
- (ii)
- \(\rank A = \tr A\);
- (iii)
- \(I - A\) is symmetric and idempotent, with \(A(I - A) = 0\) and \(\rank (I - A) = n - \rank A\);
- (iv)
- \(A\) is psd.
Proof. (i) If \(Av = \lambda v\) with \(v \neq 0\), then \(\lambda v = Av = AAv = \lambda Av = \lambda ^2v\), so \(\lambda ^2 = \lambda \) and \(\lambda \in \{0, 1\}\).
(ii) By Theorem 5.1.3 the rank is the number of non-zero eigenvalues, which by (i) is the number of eigenvalues equal to one. That is their sum, and the sum of the eigenvalues is \(\tr (Q\Lambda Q') = \tr (\Lambda Q'Q) = \tr \Lambda \).
(iii) \((I - A)' = I - A\) and \((I - A)^2 = I - 2A + A^2 = I - A\). Also \(A(I - A) = A - A^2 = 0\). By (ii), \(\rank (I - A) = \tr (I - A) = n - \tr A = n - \rank A\).
(iv) By (i) all eigenvalues are non-negative; apply Corollary 5.1.4(i). □
Two idempotent matrices appear constantly.
Example 5.1.6. The centring matrix \(M_0 = I_n - \frac 1n\iota \iota '\) subtracts the mean: \(M_0x = x - \bar x\iota \), the vector of deviations. It is symmetric, and since \(\iota '\iota = n\), \[ M_0^2 = I - \tfrac 2n\iota \iota ' + \tfrac {1}{n^2}\iota (\iota '\iota )\iota ' = I - \tfrac 1n\iota \iota ' = M_0 . \] Its trace is \(n - \frac 1n\tr (\iota '\iota ) = n - 1\), so \(\rank M_0 = n - 1\). The sum of squared deviations is a quadratic form in it: \(\sum (x_i - \bar x)^2 = (M_0x)'(M_0x) = x'M_0x\).
Example 5.1.7. Let \(X\) be an \(n \times k\) matrix of rank \(k\), so that \(X'X\) is non-singular. The matrices \[ P = X(X'X)^{-1}X', \qquad M = I_n - P \] are symmetric and idempotent: \[ P^2 = X(X'X)^{-1}(X'X)(X'X)^{-1}X' = P , \] and \(M\) is idempotent by Theorem 5.1.5(iii). By Lemma 5.1.2, \[ \tr P = \tr \bigl ((X'X)^{-1}X'X\bigr ) = \tr I_k = k, \qquad \tr M = n - k . \] So \(\rank P = k\) and \(\rank M = n - k\). Also \(PX = X\) and \(MX = 0\): \(P\) leaves the columns of \(X\) unchanged and \(M\) annihilates them. In regression \(P\) is the “hat” matrix that produces fitted values, \(\hat Y = PY\), and \(M\) the “residual maker”, \(e = MY\); both are used throughout the next section.
Differentiating linear and quadratic forms
Least squares and maximum likelihood are minimisation problems in a vector of parameters, so derivatives with respect to vectors are needed. For a scalar function \(f(\beta )\) of \(\beta = (\beta _1, \dots , \beta _k)'\), \(\partial f/\partial \beta \) denotes the column vector of partial derivatives.
Lemma 5.1.8. For a constant vector \(a\) and constant square matrix \(A\), \[ \frac {\partial (a'\beta )}{\partial \beta } = a, \qquad \frac {\partial (\beta 'A\beta )}{\partial \beta } = (A + A')\beta , \] which is \(2A\beta \) when \(A\) is symmetric. The matrix of second derivatives of \(\beta 'A\beta \) is \(A + A'\).
Proof. \(a'\beta = \sum _j a_j\beta _j\), whose derivative with respect to \(\beta _m\) is \(a_m\). And \(\beta 'A\beta = \sum _i\sum _j a_{ij}\beta _i\beta _j\); differentiating with respect to \(\beta _m\) picks up the terms with \(i = m\) and those with \(j = m\), giving \(\sum _j a_{mj}\beta _j + \sum _i a_{im}\beta _i\), the \(m\)th element of \(A\beta + A'\beta \). Differentiating again with respect to \(\beta _l\) gives \(a_{ml} + a_{lm}\). □
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.