4.6 Practice Problems
Problem 4.1. Let \[\Sigma = \begin {pmatrix} 5 & 2\\ 2 & 2 \end {pmatrix}.\] Find the principal components, their variances, and the proportion of total variation explained by the first. Where to start: solve \(\left |\Sigma -\lambda I\right |=0\) for the eigenvalues, then find a unit eigenvector for each.
Show solution
Solution. The characteristic equation is \((5-\lambda )(2-\lambda )-4 = \lambda ^{2}-7\lambda +6 = 0\), giving \(\lambda _1 = 6\) and \(\lambda _2 = 1\).
For \(\lambda _1=6\): \((\Sigma -6I)\underline {e}=0\) gives \(-e_1+2e_2=0\), so \(\underline {e}_1 \propto (2,1)'\) and, normalised, \(\underline {e}_1 = (2,1)'/\sqrt 5\). Similarly \(\underline {e}_2 = (-1,2)'/\sqrt 5\).
Hence \(Y_1 = \left (2X_1+X_2\right )/\sqrt 5\) with variance \(6\), and \(Y_2 = \left (-X_1+2X_2\right )/\sqrt 5\) with variance \(1\). The first component explains \(6/(6+1) = 0.857\) of the total variation, and \(\operatorname {tr}(\Sigma ) = 5+2 = 7 = \lambda _1+\lambda _2\) as Theorem 4.4 requires.
Problem 4.2. For the same \(\Sigma \), compute \(\operatorname {corr}(Y_1,X_1)\) and \(\operatorname {corr}(Y_1,X_2)\), and interpret the first component.
Problem 4.3. Show that if \(\Sigma \) is diagonal then the principal components are the original variables, in decreasing order of variance. Explain why this is the expected answer. Where to start: the eigenvectors of a diagonal matrix are the coordinate axes.
Problem 4.4. Let \(X_1\) be a length in centimetres and \(X_2\) a mass in grams, with \[\Sigma = \begin {pmatrix} 400 & 10\\ 10 & 4\end {pmatrix}.\] Find the first principal component from \(\Sigma \), then re-express \(X_1\) in metres and find it again. Comment on what the analysis has discovered in each case, and repeat the exercise using the correlation matrix.
Show solution
Solution. From \(\Sigma \) the first component is almost exactly \(X_1\): the variance of \(X_1\) is a hundred times that of \(X_2\) purely because of its units, so \(\lambda _1\approx 400\) with \(\underline {e}_1\approx (1,0)'\). Re-expressing the length in metres divides its variance by \(10^{4}\), giving \(\Sigma ^{*} = \begin {pmatrix}0.04 & 0.1\\ 0.1 & 4\end {pmatrix}\), and now the first component lies almost along \(X_2\). The two analyses disagree completely, and neither has found anything but the choice of unit.
From the correlation matrix, \(\rho = \begin {pmatrix}1 & 0.25\\ 0.25 & 1\end {pmatrix}\) in both cases, with \(\lambda _1 = 1.25\), \(\underline {e}_1 = (1,1)'/\sqrt 2\), and \(\lambda _2 = 0.75\). The answer is now the same whichever units are used, and it says something about the data rather than about the measuring instrument.
Problem 4.5. Prove that \(\prod ^{p}_{i=1}\lambda _i = \left |\Sigma \right |\), and hence that the generalised variance of Section 3.4 is the product of the component variances. What does a zero eigenvalue imply about the variables?
Problem 4.6. Explain why the sample principal components are unchanged if every observation has a fixed vector \(\underline {c}\) added to it, but change if each variable is multiplied by a different constant.
Problem 4.7. A scree plot of eight components shows eigenvalues \(4.4,\ 2.6,\ 0.85,\ 0.62,\ 0.48,\ 0.36,\ 0.26,\ 0.18\) obtained from a correlation matrix. How many components would each of the four criteria of this section retain? Where they disagree, say which you would follow and why. Where to start: the eigenvalues sum to \(p\); check that they do.
Problem 4.8. Give an example, or describe one, in which the first principal component conceals the difference between two known groups. What does this show about using principal components as a preliminary to classification?
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.