7.8 Practice Problems
Problem 7.1. Fisher’s linear discriminant maximises the ratio \[\frac {\left [\underline {a}'\left (\overline {\underline {x}}_1-\overline {\underline {x}}_2\right )\right ]^{2}} {\underline {a}'S_{\text {pooled}}\,\underline {a}}.\] Show that the maximum is attained at \(\underline {a}\propto S_{\text {pooled}}^{-1} \left (\overline {\underline {x}}_1-\overline {\underline {x}}_2\right )\). Where to start: the same weighted-average argument as Theorem 4.4, applied after the substitution \(\underline {b} = S_{\text {pooled}}^{1/2}\underline {a}\).
Problem 7.2. Show that the maximised value of that ratio is the squared Mahalanobis distance between the two sample means, and explain why this makes it a natural measure of separation.
Problem 7.3. Explain why a discriminant analysis and a principal component analysis of the same data can point in quite different directions, and construct a two-dimensional example in which the first principal component is useless for separating two known groups. Where to start: let the groups differ along the direction of least variation.
Show solution
Solution. Take two groups whose members are spread widely along \(x_1\) and only slightly along \(x_2\), with the two group means differing only in \(x_2\). The first principal component is then almost exactly \(x_1\), along which the groups completely overlap; the discriminant is almost exactly \(x_2\), along which they separate cleanly.
The moral is the one stated in Section 4.5: principal components maximise variance, and variance is not separation. Using the leading components as a preliminary to classification can discard precisely the direction that classifies.
Problem 7.4. State Fisher’s classification rule for two populations in terms of the discriminant score, and show that with equal prior probabilities and equal misclassification costs it assigns \(\underline {x}\) to the population whose mean is nearer in Mahalanobis distance.
Problem 7.5. The apparent error rate is computed by classifying the same observations used to fit the rule. Explain why it is optimistically biased, and describe two ways of obtaining a more honest estimate.
Show solution
Solution. The rule is chosen to separate the observations in hand, so it is fitted partly to their noise as well as to the structure; scored on those same observations it is being credited for recognising cases it has already seen. The bias grows with the number of variables relative to the sample size.
Two remedies. A holdout sample fits the rule on part of the data and scores it on the rest, which is unbiased but wastes observations. Cross-validation, in the leave-one-out form, classifies each observation using a rule fitted without it, which uses every observation and is nearly unbiased at the cost of \(n\) fits. This is the same concern, and the same remedy, as the honest accuracy discussed for any fitted classifier.
Problem 7.6. For more than two populations, explain how the discriminant directions arise as eigenvectors of \(W^{-1}B\), and state how many non-zero eigenvalues there can be when there are \(g\) groups and \(p\) variables.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.