4.5 Multicollinearity

Recall that \(Y=XB+\varepsilon \), where the columns of \(X\) are “independent” or regression variables when the columns of \(X\) are orthogonal then \(X^tX\) is a diagonal matrix. Unfortunately in most multiple regression applications the columns of \(X\) are not orthogonal in some cases when the column of \(X\) are not orthogonal there can be serious problems when there is near dependency between the independent variables columns of \(X\) matrix the problem of multi Id linearity is said to exist: \(X^t\) is about singular i.e \(\det \big (X^tX\big )\approx 0\).

Let \(X^*=\) the matrix in which the column have been centred and standardised. \[X^*_{ij}=\frac {X_{ij}-\overline {X}_j}{\sqrt {S^2_j}}\,, \quad S^2_j=\sum ^n_{j=1}\big (X_{ij}-\overline {X}_j\big )^2\]

diagonal \(X_j^{*t}X^{*}_j=\frac {\sum \limits ^n_{j=1}\big (X_{ij}-\overline {X}_{. j}\big )^2}{S^2_j}=1\)

off diagonal \begin {align*} X^{*}_jX^*_{j'} & = \frac {\sum \limits ^n_{i=1}\big (X_{ij}-\overline {X}_{.j}\big )\big (X_{ij'}-\overline {X}_{.j'}\big )}{S_jS_{j'}}\\ & = \frac {\sum \big (X_{ij}-\overline {X}_{.j}\big )\big (X_{ij'}-\overline {X}_{.j'}\big )}{\sqrt {\sum \big (X_{ij}-\overline {X}_{.j}\big )^2\sum \big (X_{ij'}-\overline {X}_{.j'}\big )^2}} \end {align*}

\(X^{*t}X^*\) the main diagonal elements are ones and the off diagonal elements are correlation coefficients i.e \(\big (j,j'\big )^{\text {th}}\) element \(\big (j\neq j'\big )\) is \(X_{ij}=\) correlation coefficients are “large” then there might be a problem. If \(X_j\) and \(X_{j'}\) are nearly dependent the \(|X_{ij}|\) \(|X_{ij'}|\) will be near unity (one).

A different way of examining multicollinearity is finding the eigenvalues of \(X^tX[|X^tX-\lambda I|]=0\) (characteristic roots).

The eigenvalues of \(X^tX\) say \(\lambda _1,\lambda _2,\ldots ,\lambda _k\) can be used to measure the extent of collinearity.

Let \(\lambda _{\max }=\max \{\lambda _1,\lambda _2,\ldots ,\lambda _k\}\) and \(\lambda _{\min }=\min \{\lambda _1,\lambda _2,\ldots ,\lambda _k\}\) then the ratio \(\frac {\lambda _{\max }}{\lambda _{\min }}\) is known as the condition number if the condition number

\[ \begin {pmatrix} n_1 & 0 & 0\\ 0 & n_2 & 0\\ 0 & 0 & n_3\\ \end {pmatrix} \] \begin {align*} \Big |X^tX-\lambda I_3\Big | & = \begin {vmatrix} n_1-\lambda & 0 & 0\\ 0 & n_2-\lambda & 0\\ 0 & 0 & n_3-\lambda \\ \end {vmatrix} =0\\ & = \big (n_1-\lambda \big )\big (n_2-\lambda \big )\big (n_3-\lambda \big )=0 \end {align*}

\[\implies \quad \lambda _1= n_1,\quad \lambda _2=n_2,\quad \lambda _3=n_3\]

Why should we be concerned about collinearity?

Recall that \(\widehat {B}=\big (X^tX\big )^{-1}X^tY\) and \(\quad Cov\big (\widehat {B}\big )=\big (X^tX\big )^{-1}\sigma ^2\).

The least squares estimation produces estimated regression coefficients that are too large in absolute value.

There are a number of methods of dealing with collinearity.

• Collecting additional data
• Model respecification
• Ridge regression, \(\big (X^tX+\alpha I\big )\widehat {B}_n=X^tY\) \(\{\) becomes your normal equations \(\}\)
• Generalised ridge regression
• Principal component regression \[Y=Z\alpha + \varepsilon ,\quad Z=XT,\] where \(T\) is the matrix whose columns are the eigenvectors of \(\lambda _1,\lambda _2,\ldots ,\lambda _k\).
• Latent root regression analysis.
  

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.