4.3 Components from the Correlation Matrix

Principal components are not invariant under rescaling of the variables, and this is the most common source of error in applying them.

Note 4.8. Suppose one variable is a length in millimetres and another a mass in kilogrammes. The first has a numerically large variance purely because of its units, so the first component will point almost entirely along it and the analysis will have discovered nothing but the choice of scale. Re-expressing the length in metres would give a completely different answer.

The remedy is to standardise: work with \(Z_k = \left (X_k-\mu _k\right )/\sqrt {\sigma _{kk}}\), whose covariance matrix is the correlation matrix \(\rho \). Components extracted from \(\rho \) are unaffected by the units, and \(\operatorname {tr}(\rho )=p\), so the proportion explained by the \(i\)th component is simply \(\lambda _i/p\).

Components from \(\Sigma \) and from \(\rho \) are not related in any simple way — one cannot be obtained from the other by rescaling. Which to use is a decision that must be made and stated. Standardise unless the variables are already in comparable units and their differing variances are themselves meaningful.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.