9.4 Practice Problems
Problem 9.1. Write the multivariate regression model \(Y = ZB + \varepsilon \) in full, identifying the dimensions of every matrix, and state the assumptions on \(\varepsilon \).
Problem 9.2. Show that the least squares estimator is \(\widehat {B} = \left (Z'Z\right )^{-1}Z'Y\), and that each column of \(\widehat {B}\) is exactly the univariate least squares estimator obtained by regressing the corresponding response on \(Z\) alone. Where to start: minimise \(\operatorname {tr}\left [(Y-ZB)'(Y-ZB)\right ]\), and note that it separates into a sum over the columns.
Show solution
Solution. Writing the trace as a sum over responses, \[\operatorname {tr}\left [(Y-ZB)'(Y-ZB)\right ] = \sum ^{m}_{i=1}\left (\underline {y}_{(i)}-Z\underline {b}_{(i)}\right )' \left (\underline {y}_{(i)}-Z\underline {b}_{(i)}\right ),\] in which the \(i\)th term involves only the \(i\)th column of \(B\). The sum is therefore minimised by minimising each term separately, which is the ordinary least squares problem for that response, giving \(\widehat {\underline {b}}_{(i)} = (Z'Z)^{-1}Z'\underline {y}_{(i)}\) and hence \(\widehat {B} = (Z'Z)^{-1}Z'Y\).
This is worth noticing: the point estimates gain nothing from being computed jointly. What the multivariate treatment adds is the covariance structure between responses, which is what tests and confidence regions depend on.
Problem 9.3. Show that the residual matrix \(\widehat {\varepsilon } = Y - Z\widehat {B}\) satisfies \(Z'\widehat {\varepsilon }=0\), and interpret this geometrically.
Problem 9.4. Given the result of the second problem, explain why a multivariate regression is nevertheless worth performing rather than \(m\) separate regressions.
Problem 9.5. Show that \(\widehat {\Sigma } = \tfrac {1}{n}\widehat {\varepsilon }'\widehat {\varepsilon }\) is biased and that dividing instead by \(n-r-1\), where \(r+1\) is the number of columns of \(Z\), removes the bias.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.