3.3 Summary Statistics

What follows are basic summary statistics and some standard statistics.

3.3.1 The Mean Vector and Covariance Matrix

An obvious extension of the uni variate notion of mean and variance leads to the following definitions.
The sample mean of the \(i^{\text {th}}\) variable is \(\displaystyle {\overline {X}_i=\frac {1}{n}\sum ^n_{r=1}x_{ri}\qquad (1)}\,\) and the sample variance of the \(i^{\text {th}}\) variable is \[S_{ii}=\frac {1}{n}\sum ^n_{i=1}\big (x_{ri}-\overline {X}_{i}\big )^2=S^2_i\qquad (2)\qquad i= 1,2,\ldots , p\] the sample Covariance between the \(i^{\text {th}}\) and \(j^{\text {th}}\) variable is \begin {equation} \tag {3} S_{ij} = \frac {1}{n}\sum ^n_{r=1}\big (x_{ri}-\overline {X}_i\big )\big (x_{rj}-\overline {X}_j\big ) \end {equation} The vector of means \(\displaystyle {\overline {\underline {X}}= \begin {pmatrix} \overline {X}_1\\ \overline {X}_2\\ \vdots \\ \vdots \\ \overline {X}_p \end {pmatrix} }\,\) is called the sample mean vector or simply ‘mean vector’.

The \(p\times p\) matrix \(S=(S_{ij})\qquad (5)\,\) with elements given by (2) and (3) is called the sample Covariance matrix or simply the Covariance matrix.

The above statistics may also be expressed in the matrix notation corresponding to (4) and (5) we have

\begin {align*} \overline {\underline {X}} & = \frac {1}{n}\sum ^n_{r=1}\underline {X}_r= \frac {1}{n}\Bigg \{ \begin {pmatrix} x_{11}\\ \vdots \\ \vdots \\ x_{1p}\\ \end {pmatrix} + \begin {pmatrix} x_{21}\\ \vdots \\ \vdots \\ x_{2p}\\ \end {pmatrix} + \cdots + \begin {pmatrix} x_{n1}\\ \vdots \\ \vdots \\ x_{np}\\ \end {pmatrix} \Bigg \}\\\\ & = \frac {1}{n} \begin {pmatrix} x_{11} + x_{21} +\cdots + x_{n1}\\ x_{12} + x_{22} + \cdots + x_{n2}\\ \vdots \\ \vdots \\ x_{1p} + x_{2p} + \cdots + x_{np}\\ \end {pmatrix}\\\\ & = \frac {1}{n}X'\underline {1}_n\tag {6} \end {align*}

where \(\underline {1}\) is a column vector of \(n\) ones \((1)\).

Also \(\displaystyle {S_{ij}=\frac {1}{n}\sum \big (x_{ri}-\overline {X}_i\big )\big (x_{rj}-\overline {X}_j\big ) = \frac {1}{n}\sum ^n_{r=1}x_{ri}x_{rj}-\overline {X}_i\overline {X}_j}\)

So that \begin {align*} S & = \frac {1}{n}\sum ^n_{r=1}\big (\underline {X}_r-\overline {\underline {X}}\big )\big (\underline {X}_r-\overline {\underline {X}}\big )'\\ & = \frac {1}{n}\sum ^n_{r=1} \underbrace {\underline {X}_r\underline {X}'_r}_{p\times 1 \cdot 1\times p}-\underbrace {\overline {\underline {X}}\,\overline {\underline {X}}'}_{p\times 1 \cdot 1\times p}\tag {6} \end {align*}

\(\qquad \displaystyle {\underline {X}_1=\binom {x_{11}}{x_{21}}}\qquad \underline {X}_2=\displaystyle {\binom {x_{21}}{x_{22}}}\)

\begin {align*} S & = \frac {1}{2}\sum ^2_{r=1}\big (\underline {X}_r-\underline {\overline {X}}\big )\big (\underline {X}_r-\overline {\underline {X}}\big )'\\ & = \frac {1}{2}\Bigg \{ \begin {pmatrix} x_{11}-\overline {X}_1\\ x_{21}-\overline {X}_2\\ \end {pmatrix} \begin {pmatrix} x_{11}-\overline {X}_1 & x_{12}-\overline {X}_2\\ \end {pmatrix} + \begin {pmatrix} x_{21}-\overline {X}_1\\ x_{22}-\overline {X}_2\\ \end {pmatrix} \begin {pmatrix} x_{21}-\overline {X}_1 & x_{22}-\overline {X}_2\\ \end {pmatrix} \Bigg \}\\\\ & = \frac {1}{2}\Bigg \{ \begin {bmatrix} \big (x_{11}-\overline {X}_1\big )^2 & \big (x_{11}-\overline {X}_1\big )\big (x_{12}-\overline {X}_2\big )\\ \big (x_{12}-\overline {X}_2\big )\big (x_{11}-\overline {X}_1\big ) & \big (x_{12}-\overline {X}_2\big )\\ \end {bmatrix}\\ & + \begin {bmatrix} \big (x_{21}-\overline {X}_1\big )^2 & \big (x_{21}-\overline {X}_1\big )\big (x_{22}-\overline {X}_2\big )\\ \big (x_{22}-\overline {X}_2\big )\big (x_{21}-\overline {X}_1\big ) & \big (x_{22}-\overline {X}_2\big )^2\\ \end {bmatrix} \Bigg \}\\\\ & = \begin {pmatrix} \frac {\big (x_{11}-\overline {X}_1\big )^2+\big (x_{21}-\overline {X}_1\big )^2}{2} & \frac {\big (x_{11}-\overline {X}_1\big )\big (x_{12}-\overline {X}_2\big )+\big (x_{21}-\overline {X}_1\big )\big (x_{22}-\overline {X}_2\big )}{2}\\\\ \frac {\big (x_{12}-\overline {X}_2\big )\big (x_{11}-\overline {X}_1\big )+\big (x_{22}-\overline {X}_2\big )\big (x_{21}-\overline {X}_1\big )}{2} & \frac {\big (x_{12}-\overline {X}_2\big )^2+\big (x_{22}-\overline {X}_2\big )}{2}\\ \end {pmatrix} \end {align*}

\[\therefore \quad S= \begin {pmatrix} S_{11} & S_{12}\\ S_{21} & S_{22}\\ \end {pmatrix} \]

\(S\) may also be written as \begin {align*} S & = \frac {1}{n}X'X-\overline {\underline {X}}\,\overline {\underline {X}}'\\ & = \frac {1}{n}\bigg (X'X-\frac {1}{n}X'\underline {1}\,\underline {1}'X\bigg )\\ & = \frac {1}{n}X'\bigg (I-\frac {1}{n}\underline {1}\,\underline {1}\bigg )X\\ & = \frac {1}{n}X'HX\tag {8} \end {align*}

where \(\displaystyle {H=I-\frac {1}{n}\underline {1}\,1'}\qquad (9)\)
\(H\) was encountered earlier and we called it the centering matrix.
(8) is the other way of representing the sample Covariance.
Recall that the \(H\) matrix has the following properties:

1.

it is symmetric

2.

idempotent matrix

has in one-dimension statistics it is often convenient to define the Covariance matrix with a divisor \(n-1\) instead of \(n\). This Covariance matrix is \begin {equation} \tag {10} S_n= \frac {n-1}{n}S_{n-1}=\frac {1}{n-1}X'HX \end {equation} If the data forms a random sample from multivariate variable distribution with finite second moments, then \(S_n\) is unbiased estimator of the true Covariance matrix.

The matrix \begin {equation} \tag {11} M= \sum ^n_{r=1} \underline {X}_r\underline {X}_r'=X'X\, \cdot \end {equation} is called the matrix of sums of squares and cross products.
See the previous example.

\(\bullet \) The matrix \(nS\) or \((n-1)S_n\) is the matrix of collected sums of squares and cross products.

3.3.2 Correlation Matrix

The sample correlation coefficient between the \(i^{\text {th}}\) and \(j^{\text {th}}\) variables is \begin {equation} \tag {12} r_{ij}=\frac {S_{ij}}{S_iS_j}=\frac {S_{ij}}{\sqrt {S_{ii}}\sqrt {S_{jj}}} \end {equation}

The matrix \begin {equation} \tag {13} R=(r_{ij}) \end {equation} with \(r_{ij}=1\) is called the sample correlation matrix.

1.

\(R\geq 0\)

2.

If \(R=I\), we say that the variables are uncorrelated.

3.

If \(D=diag(S_i)\) then \(R=D^{-1}SD^{-1}\), \(\, S=DRD\qquad (14)\)

\begin {align*} R & = D^{-1}SD^{-1}\\ & = \begin {pmatrix} \frac {1}{3} & 0\\ 0 & \frac {1}{2}\\ \end {pmatrix} \begin {pmatrix} 9 & 2\\ 2 & 4\\ \end {pmatrix} \begin {pmatrix} \frac {1}{3} & 0\\ 0 & \frac {1}{3}\\ \end {pmatrix}\\ & = \begin {pmatrix} 3 & \frac {2}{3}\\ 1 & 2\\ \end {pmatrix} \begin {pmatrix} \frac {1}{3} & 0\\ 0 & \frac {1}{2}\\ \end {pmatrix}\\ & = 1\\ \end {align*}