2.3 Derivatives of Linear and Quadratic Forms

Least squares minimises \(\left (\underline {Y}-XB\right )^t \left (\underline {Y}-XB\right )\) over \(B\). Doing this by differentiating with respect to each \(\beta _j\) in turn gives \(k\) equations to be assembled by hand; doing it in matrix form gives the normal equations in three lines.

Only two derivatives are needed, and they are the matrix analogues of \(\frac {d}{dx}(ax)=a\) and \(\frac {d}{dx}(ax^2)=2ax\). They are stated and proved here, and applied in Section 3.3.

Definition 2.3.1. Derivatives of a function with respect to a vector. Let \(f(X)\) be a function of \(k\) “independent” real variables. \(X= \begin {pmatrix} x_1 & x_2 & \cdots & x_k\\ \end {pmatrix}^t \). The derivative of \(f(X)\) with respect to \(X\) is denoted by \(\frac {\partial }{\partial X}f(X)\) and is defined by \[\frac {\partial }{\partial X}f(X)= \begin {pmatrix} \frac {\partial }{\partial x_1}f(X) & \frac {\partial }{\partial x_2}f(X) &\cdots & \frac {\partial }{\partial x_k}f(X)\\ \end {pmatrix}^t \]

Example 2.3.2. Consider a function of real variables \(x_1,x_2\) and \(x_3\).

\begin {align*} f(X) & = 6x_1^2+x_2^2-2x_1x_2+x_2x_3+2x_3^2\\ \frac {\partial f(X)}{\partial x_1} & = 12x_1-2x_2,\quad \frac {\partial f(X)}{\partial x_2}=2x_2-2x_1+x_3,\quad \frac {\partial f(X)}{\partial x_3}=x_2+4x_3\\ \therefore \quad \frac {\partial f(X)}{\partial X} & = \begin {pmatrix} 12x_1-2x_2 & 2x_2-2x_1+x_3 & x_2+4x_3 \end {pmatrix}^t\\ \end {align*}

Result 2.3.3.

i).: Let \(L(X)\) be a linear function of \(k\) “independent” real variables given by \[L(X)=a^tX=\sum ^k_{j=1}a_jx_j\] \[\frac {\partial L(X)}{\partial X}=a\qquad ,\qquad a= \begin {pmatrix} a_1 & a_2 & \cdots & a_k\\ \end {pmatrix}^t \]
ii).: Let \(Q(X)=X^tAX\), where \(A\) is a matrix of constants then \[\frac {\partial Q(X)}{\partial X}=2AX\]

Example 2.3.4. Let \(Q(X)=6x_1^2+x^2_2-2x_1x_2+x_2x_3+2x^2_3= \begin {pmatrix} x_1 & x_2 & x_3\\ \end {pmatrix} \begin {pmatrix} 6 & -1 & 0\\ -1 & 1 & 1/2\\ 0 & 1/2 & 2\\ \end {pmatrix} \begin {pmatrix} x_1\\x_2\\x_3\\ \end {pmatrix} \)

\begin {align*} \frac {\partial Q(X)}{\partial X} & = 2AX=2 \begin {pmatrix} 6 & -1 & 0\\ -1 & 1 & 1/2\\ 0 & 1/2 & 2\\ \end {pmatrix} \begin {pmatrix} x_1\\x_2\\x_3\\ \end {pmatrix}\\ & =2 \begin {pmatrix} 6x_1-x_2\\ -x_1+x_2+\frac {1}{2}x_3\\ \frac {1}{2}x_2 +2x_3\\ \end {pmatrix}\\ & = \begin {pmatrix} 12x_1-2x_2\\ -2x_1+2x_2+x_3\\ x_2+4x_3 \end {pmatrix}\\ \end {align*}

Definition 2.3.5.

i.: An \(m\times m\) matrix \(B\) is defined to be idempotent if \(B^2=B\). If \(B\) is also symmetric then \(B\) is said to be symmetric idempotent matrix.
ii.: Traces of an \(m\times m\) matrix \(B\), denoted by \(tr(B)\) is the sum of the diagonal elements of \(B\). \[\text {i.e}\quad tr(B)=\sum ^n_{j=1}b_{ij}\]

Theorem 2.3.6. Let \(B\) be an \(m\times m\) idempotent matrix

i.: If \(rank(B)=m\), then \(B=I\).
ii.: If \(B\) is symmetric and \(P\) is an \(m\times m\) orthogonal matrix then \(P^tBP\) is symmetric idempotent matrix.
iii.: If \(P\) is non-singular matrix and \(B\) is symmetric idempotent matrix then \(PBP^{-1}\) is idempotent matrix.
iv.: If \(B\) is symmetric idempotent matrix then \(I-B\) is also symmetric idempotent matrix.

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.