2.2 Expectation of Random Vectors and Variance-Covariance

A random vector needs a mean and a measure of spread, and for a vector the second of these is a matrix: variances along the diagonal, covariances off it. The definitions below are the matrix versions of familiar scalar ones, and the results that follow answer the question that recurs throughout the course — if \(\underline {Y}\) has mean \(\underline {\mu }\) and variance \(\Sigma \), what are the mean and variance of \(A\underline {Y}\)?

That single question, with different choices of \(A\), produces the distribution of \(\widehat {B}\), of the fitted values, and of the residuals.

Definition 2.2.1.

i.: Let \(Y\) be a vector of random variables i.e \(Y= \begin {pmatrix} Y_1 & Y_2 & \cdots & Y_k\\ \end {pmatrix}^t \) and \(Y_{j'}s\) are random variables, then \(Y\) is a random vector.
ii.: Let a matrix \(V_{k\times m}\) of jointly distributed random variables i.e \(V_{ij}\) is a random variable, where \(i=1,2,\ldots , k\quad j=1,2,\ldots ,m\) then \(V\) is a random matrix.

Definition 2.2.2.

i.: Given a random vector \(Y= \begin {pmatrix} Y_1 & Y_2 & \cdots & Y_k\\ \end {pmatrix}^t \) then the expectation of \(Y\) denoted by \(E(Y)\) is given by \[E(Y)= \begin {pmatrix} E(Y_1) & E(Y_2) & \cdots & E(Y_k)\\ \end {pmatrix}^t \]
ii.: Given a random matrix \(V_{k\times m}\), then the expectation of \(V\) denoted by \(E(V)\) is given by \[ E(V)=\Big (E(V_{ij})\Big )_{k\times m}= \begin {pmatrix} E(V_{11}) & E(V_{12}) & \cdots & E(V_{1m})\\ E(V_{21}) & E(V_{22}) & \cdots & E(V_{2m})\\ \vdots & \vdots & & \vdots \\ E(V_{k1}) & E(V_{k2}) & \cdots & E(V_{km})\\ \end {pmatrix} \]

Lemma 2.2.3. If \(A\) and \(B\) are matrices of constants and \(V\) is a random matrix the \[E\big (AVB\big )=AE\big (V\big )B.\]

Proof. □

Definition 2.2.4. Consider a random vector \(Y= \begin {pmatrix} Y_1 & Y_2 & \cdots & Y_k\\ \end {pmatrix}^t \), with \(Y_{j'}\)s having finite variances

i.: Let \(\mu _j=E(Y_j),\quad j=1,2,\ldots ,k\) then \( \mu = \begin {pmatrix} \mu _1 & \mu _2 & \cdots & \mu _k \end {pmatrix}^t \), \(\mu \) is the mean vector of \(Y\). i.e \(\mu =E(Y)\).
ii.: The matrix \(\Sigma _Y\) whose \((i,j)^{\text {th}}\) element is the covariance of \(Y_i\) and \(Y_j\) is called the variance-covariance matrix of \(Y\) and is given by \[\underbrace {\Sigma _Y}_{k\times k} = E\Big \{\underbrace {\big (Y-\mu \big )}_{k\times 1}\underbrace {\big (Y-\mu \big )^t}_{1\times k}\Big \}\]

Lemma 2.2.5. For any linear transformation \(Y=AX\) from \(k\) random variables \(X_1,X_2,\ldots ,X_k\) with matrix \(A_{m\times k}\) we have

i.: \(\mu _Y=E(Y)=AE(X)=A\mu _X\)
ii.: \(\Sigma _Y=Cov(Y)=A\Sigma _XA^t,\quad \Sigma _X=Cov(X)\)

Proof.

i.: Exercise
ii.: \begin {align*} \Sigma _Y & = E\Big \{(Y-\mu _Y)(Y-\mu _Y)^t\Big \}\\ & = E\Big \{(AX-A\mu _X)(AX-A\mu _X)^t\Big \}\\ & = E\Big \{A(X-\mu _X)(A(X-\mu _X))^t\Big \}\\ & = E\Big \{A(X-\mu _X)(X-\mu _X)^tA^t\Big \}\\ & = AE\Big \{(X-\mu _X)(X-\mu _X)^t\Big \}A^t\\ & = A\Sigma _XA^t \end {align*}

Remark. The variance-covariance matrix of any Random vector is symmetric.

□

Theorem 2.2.6. If \(Y\) is a random vector with \(k\) components with mean vector \(\mu \), covariance matrix \(\Sigma _Y\) which is non-singular and each component of \(Y\) has normal distribution then \[Q(Y) = (Y-\mu _Y)^t\Sigma ^{-1}_Y(Y-\mu _Y)\] has a Chi-square distribution with \(k-\) degrees of freedom.

Proof. Since \(\Sigma _Y\) is symmetric then there exists a non-singular matrix \(P\) such that \begin {align*} P\Sigma _YP^t & = I\\ \implies \quad P^tP\Sigma _YP^tP & = P^tIP\\ P^tP\Sigma _YP^tP & = P^tP\\ \Sigma _Y & = \big (P^tP\big )^{-1}\\ \implies \quad \Sigma ^{-1}_Y & = P^tP\\ \end {align*}

Define a transformation \[\underbrace {W}_{k\times 1}=\underbrace {P}_{k\times k} \underbrace {(Y-\mu _Y)}_{k\times 1}\] \begin {align*} W^tW & = \Big (\big (P(Y-\mu _Y)\big )^t\big (P(Y-\mu _Y)\big )\Big )\\ & = (Y-\mu _Y)^tP^tP(Y-\mu _Y)\\ & = (Y-\mu _Y)^t\Sigma ^{-1}_Y(Y-\mu _Y) \end {align*}

\[W^tW=\sum ^k_{i=1}W_i^2,\quad W_i\quad E(W)=0\] \[Cov(W)=E\Big (P(Y-\mu _Y)\big (P(Y-\mu _Y)\big )^t\Big )\]

\(\mu \) is the \(PE\Big ((Y-\mu _Y)(Y-\mu _Y)^t\Big )P^t\)

\begin {align*} Cov(W) & = P\Sigma _YP^t\\ & = P\big (P^tP\big )^{-1}P^t\\ & = PP^{-1}\big (P^t\big )^{-1}P^t\\ & = I \end {align*}

Hence the Proof. □

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.