3.5 Geometrical Ideals

3.5.1 R-Techniques

If the interest is to compare columns of the data matrix the techniques to use are called \(R-\) techniques because the correlation matrix \(R\) plays an important role in this approach.
\(R-\) techniques are important in principal component analysis, factor analysis and canonical correlation.

3.5.2 Q-Techniques

Sometimes there is need to compare rows of the data matrix, the different objects. This leads to techniques such as discrete, cluster analysis and multi dimensional scaling which are known as \(Q-\) techniques.
These two approaches correspond to different geometrical ways representing \(n\times p\) matrix.
The columns are viewed as \(p\) points in a \(n\)- dimensional space called the \(R\) space. The correlation \(r_{ij}\) between the variable \(i\) and \(j\) are viewed as Cosines of the angle \(\theta _{ij}\) made at the original \[\cos \theta _{ij}=\frac {Y_iY_j}{||Y_i||||Y_j||}=\frac {S_{ij}}{S_iS_j}=r_{ij}\] The rows may be viewed as \(n\) points in \(p-\) dimension \(Q-\) space of variable space.
Natural way to compare two rows \(\underline {X}_r\) and \(\underline {X}_s\) is to look at the Euclidean distance between them \[||\underline {X}_r-\underline {X}_s||^2=\big (\underline {X}_r-\underline {X}_s\big )'\big (\underline {X}_r-\underline {X}_s\big )\] An alternative procedure is to transform the data by one of the many transformations that exists and then exam the Euclidean distance between the transformed row.

Cluster analysis. The most important of these distances is the MAHALANOBI’S Distance \(D_{rs}\) given by \[D^2_{rs}=||\underline {Z}_r-\underline {Z}_s||^2=\big (\underline {X}_rS^{-1}\big (\underline {X}_r-\underline {X}_s\big )\]

3.5.3 Univariate Scatters

The need for graphical representations when \(p>3\) is greater than for the univariate case since the relationships can be understood using the data matrix.
A simple starting point is to look in the plot of \(p\) variables side by side. Thus, such plots are the dot plot and the consecutive univariate plot.

1.

Dot Plot
\(\bullet \) When the variables are measured in same unit it is possible to have a direct comparison but standardisation of the variables is helpful.
\(\bullet \) Doesn’t give any ideal of relationships of between variables but location and spread.

2.

Consecutive Plot
\(\bullet \) Plotting observations consecutively along an axis, representing different variables by different symbols helps to exhibit interrelationships.

3.

Bivariate Scatters
Another way of understanding interrelationships is looking into \(\displaystyle {\frac {p(p-1)}{2}}=\binom {p}{2}\) bivarite scatters of diagrams of the data.
Other plots exist which take advantage of new statistical softwares.

3.5.4 Linear Combinations

\(\bullet \) Taking linear combinations of the variables is one of the most important tools of multivariate analysis.
\(\bullet \) Few chosen combinations provide more information than a multiplicity of the origin variable often because of the dimensions of a vector is reduced.
\(\bullet \) Linear transformations can also simplify the structure making interpretation of the data more straight forward. \[y_r = a_1X_{r_1} + a_2X_{r_2}+\cdots + a_pX_{rp}\qquad r=1,2,\ldots , n\qquad (1)\] where \(a_1,a_2,\ldots ,a_p\) are given constants.
\(y_r\) is a linear combination of the components of the vector for the \(r^{\text {th}}\) object. \[\underline {X}_r= \begin {pmatrix} x_{r_1}\\ x_{r_2}\\ \vdots \\ \vdots \\ x_{r_p}\\ \end {pmatrix} \]

In the data matrix, we are looking at a particular row.

What are the mean and variance of \(y_r\)?

The Mean \(\displaystyle {\overline {Y}= \frac {\sum \limits ^n_{r=1}Y_r}{n}=\frac {\sum \limits ^n_{r=1}\underline {a}'\underline {X}_r}{n}=\frac {\underline {a}\sum \limits ^n_{r=1}\underline {X}_r}{n}=\underline {a}'\underline {\overline {X}}\qquad (2)}\)

The Variance \begin {align*} S^2_Y & = \frac {1}{n}\sum ^n_{r=1}\big (Y_r-\overline {Y}\big )^2 = \frac {1}{n}\sum ^n_{r=1}\big (Y_r-\overline {Y}\big )\big (Y_r-\overline {Y}\big )\\ & = \frac {1}{n}\sum ^n_{r=1}\big (Y_r-\overline {Y}\big )\big (Y_r-\overline {Y}\big )'\qquad \text {since}\quad Y_r-\overline {Y}\qquad \text {is}\quad |X|\\ & = \frac {1}{n}\sum ^n_{r=1}\big (\underline {a}'\underline {X}_r-\underline {a}'\underline {\overline {X}}\big )\big (\underline {a}'\underline {X}_r-\underline {a}'\overline {\underline {X}}\big )'\\ & = \frac {1}{n}\sum ^n_{r=1}\underline {a}'\big (\underline {X}_r-\overline {\underline {X}}\big )\big (\underline {X}_r-\overline {\underline {X}}\big )'\underline {a}\\ & = \underline {a}\frac {1}{n} \sum ^n_{r=1}\big (\underline {X}_r-\overline {\underline {X}}\big )\big (\underline {X}_r-\overline {\underline {X}}\big )'\underline {a}\\ & = \underline {a}'S\underline {a}\\ & = \underline {a}'E\lambda E'\underline {a}\\ & = \big (E'\underline {a}\big )'\lambda E'\underline {a}\\ & = \underline {W}'\lambda \underline {W}\\ & = \sum \lambda _iw_i^2 \end {align*}

In general we may be interested in a \(Q-\) dimensional linear transformation \[\underline {Y}_r=A\underline {X}_r+\underline {b}, r=1,2,\ldots , n\qquad (3)\] which may be written \(Y=XA'+\underline {1}\, \underline {b}'\) where \(A\) is a \(q\times p\) matrix and \(\underline {b}\) is a \(q-\) vector of constants. Usually \(q\leq p\).

The mean and Covariance matrix of (3) are given by

\begin {align*} \text {The Mean}\qquad \overline {\underline {Y}} & = \frac {\sum \limits ^n_{r=1}\underline {Y}_r}{n}=\frac {\sum \limits ^n_{r=1}\big (A\underline {X}_r + \underline {b}\big )}{n}\\\\ & = \frac {A \sum \limits ^n_{r=1}\underline {X}_r}{n}+ \frac {\sum \limits ^n_{r=1}\underline {b}}{n}\\\\ & = A\underline {\overline {X}} + \underline {b}\tag {4}\\ \end {align*}

\begin {align*} \text {The Covariance}\qquad S_n & = \frac {1}{n}\sum ^n_{r=1}\big (\underline {Y}_r-\overline {\underline {Y}}\big )\big (\underline {Y}_r-\overline {\underline {Y}}\big )'\\ & = \frac {1}{n}\sum ^n_{r=1}\big (A\underline {X}_r+\underline {b}-A\underline {\overline {X}}-\underline {b}\big )\big (A\underline {A}_r+\underline {b}-A\underline {\overline {X}}-\underline {b}\big )'\\ & = A \frac {1}{n}\sum ^n_{r=1}\big (\underline {X}_r-\underline {\overline {X}}\big )\big (\underline {X}_r-\overline {\underline {X}}\big )'A'\\ & = ASA'\tag {5} \end {align*}

If \(A\) is non-singular then \(\displaystyle {S=A^{-1}S_Y\big (A'\big )^{-1}\qquad (6)}\)

Examples of Linear Transformations

1.

The Scaling Transformation
Let \(Y_r=D^{-1}\big (\underline {X}_r-\underline {X}\big ).\qquad r=1,2,\ldots , n.\) where \(D=diag(S_i)\). This transformation scales each variable to have units variance and that’s eliminates the arbitrariness in the choice of scale e.g If \(\underline {X}_{(i)}\) measures length then \(\underline {Y}_{(i)}\) will be the same whether \(\underline {X}_i\) is measured in inches or meters.

2.

Mahalanobi’s Transformation (MT)
If \(S>0\) then \(S^{-1}\) has a unique symmetric positive definite, \(S^{-1/2}\).
In (MT) is defined by \[\underline {Z}_r=S^{{\frac {-1}{2}}} \big (\underline {X}_r-\overline {\underline {X}}\big ), r=1,2,\cdot \cdots ,n\] Then \(\displaystyle {S_Z=\frac {1}{n}\sum ^n_{r=1}\big (\underline {Z}_r-\overline {\underline {Z}}\big )\big (\underline {Z}_r-\overline {\underline {Z}}\big )'}\)

3.

The Principal Component Transformation (P.C.T)
By spectral decomposition theorem the Covariance matrix \(S\) may be written in the form \[S=GLG'\qquad E\lambda E'\] where \(G\) is an orthogonal matrix and \(L\) is diagonal matrix of eigenvalues \[L_1\leq L_2\leq \cdots L_p\leq 0\] The P.C.T is defined by \(\underline {W}_r=G'\big (\underline {X}_r-\overline {\underline {X}}\big ), r=1,2,\ldots ,n\). Since
\(S_W=GSG'=L\) is diagonal, the columns of \(W\) matrix, called principal components, represent uncorrelatec linear transformations of the variables.
In practice one hopes to summarise most of the variability using the P.C.T with the highest variances, thus reducing the dimension.

\begin {align*} d^2\big (X,\mu ) & = \Bigg [\frac {X-\mu }{\sigma }\Bigg ]^2 = (X-\mu ) \frac {1}{\sigma ^2}(X-\mu )\\ & = (X-\mu )\sigma ^{-2}(X-\mu )\\\\ & (\underline {X}-\mu )'\Sigma ^{-1}(\underline {X}-\mu )\\ & (\underline {X}-\mu )'E\lambda ^{-1}E'(\underline {X}-\mu ) \end {align*}

BLANK SPACE

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.