2.6 Practice Problems
Problem 2.1. Let \(\underline {X}\) have mean \(\underline {\mu }\) and covariance \(\Sigma \), and let \(\underline {Y} = A\underline {X}+\underline {b}\). Show that \(E(\underline {Y}) = A\underline {\mu }+\underline {b}\) and \(\operatorname {cov}(\underline {Y}) = A\Sigma A'\). Where to start: expand \(E\left [(\underline {Y}-E\underline {Y})(\underline {Y}-E\underline {Y})'\right ]\) and note that \(\underline {b}\) cancels.
Problem 2.2. Show that \(\operatorname {var}\left (\underline {a}'\underline {X}\right ) = \underline {a}'\Sigma \underline {a}\), and hence that \(\Sigma \) is positive semi-definite. Under what condition is it positive definite?
Show solution
Solution. The first part is the previous problem with \(A = \underline {a}'\). Since a variance cannot be negative, \(\underline {a}'\Sigma \underline {a}\geq 0\) for every \(\underline {a}\), which is the definition of positive semi-definiteness.
It fails to be positive definite exactly when some \(\underline {a}\neq \underline {0}\) has \(\operatorname {var}(\underline {a}'\underline {X}) = 0\), that is when a linear combination of the variables is constant with probability one. So \(\Sigma \) is positive definite unless the variables satisfy an exact linear relation - which is the same degeneracy that gives a zero eigenvalue in Section 4 and a singular \(S\) in Section 5.3.
Problem 2.3. If \(D = \operatorname {diag}\left (\sqrt {\sigma _{11}},\ldots ,\sqrt {\sigma _{pp}}\right )\), show that the correlation matrix satisfies \(R = D^{-1}\Sigma D^{-1}\) and \(\Sigma = DRD\). Verify that \(R\) has unit diagonal.
Problem 2.4. Two random variables have \(\operatorname {cov}(X_1,X_2)=0\) but are not independent. Construct such a pair, and state the additional assumption under which zero covariance would imply independence. Where to start: take \(X_1\) symmetric about zero and \(X_2 = X_1^{2}\).
Problem 2.5. Partition \(\underline {X}\) into \(\underline {X}_1\) and \(\underline {X}_2\) and write the covariance matrix in blocks. Show that \(\operatorname {cov}\left (\underline {X}_1,\underline {X}_2\right ) = \Sigma _{12}\) and that \(\Sigma _{21} = \Sigma _{12}'\).
Problem 2.6. For \(\underline {X}\) with covariance \(\Sigma \), show that the conditional expectation \(E\left (\underline {X}_1\mid \underline {X}_2\right )\) minimises \(E\left \|\underline {X}_1-g(\underline {X}_2)\right \|^{2}\) over all functions \(g\). Why does this matter for the regression of Section 9?
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.