10.1 The Orthogonal Factor Model
Definition 10.1 (Factor model). The vector \(\underline {X}\) with mean \(\underline {\mu }\) follows the \(m\)-factor model if \[\underline {X}-\underline {\mu } = L\,\underline {F} + \underline {\varepsilon },\] where \(L\) is the \(p\times m\) matrix of loadings, \(\underline {F}\) is the \(m\times 1\) vector of common factors and \(\underline {\varepsilon }\) the \(p\times 1\) vector of specific factors, with \[E(\underline {F}) = \underline {0},\quad \operatorname {cov}(\underline {F}) = I_m,\quad E(\underline {\varepsilon }) = \underline {0},\quad \operatorname {cov}(\underline {\varepsilon }) = \Psi = \operatorname {diag}(\psi _1,\dots ,\psi _p),\] and \(\underline {F}\) independent of \(\underline {\varepsilon }\).
Theorem 10.2 (The covariance structure). Under the factor model, \[\Sigma = LL' + \Psi ,\] so that \[\sigma _{ii} = \underbrace {\ell _{i1}^{2}+\cdots +\ell _{im}^{2}}_{\text {communality } h_i^{2}} + \underbrace {\psi _i}_{\text {specific variance}}, \qquad \sigma _{ik} = \sum ^{m}_{j=1}\ell _{ij}\ell _{kj} \ \ (i\neq k).\]
Note 10.3. The second equation is the whole content of the model, and it is a strong claim: all the covariance between distinct variables is carried by the common factors, since \(\Psi \) is diagonal and contributes nothing off the diagonal. If the model holds with small \(m\), the \(\binom {p}{2}\) covariances are generated by only \(pm\) loadings, which is a genuine and testable reduction.
The variance of each variable splits into the part explained by the common factors, its communality, and the part specific to it. A variable with low communality is not being described by the factors at all, and its presence in the analysis should be reconsidered.
Note 10.4 (Why this is not principal component analysis). Three differences, in order of importance.
Principal components are defined as linear combinations of the observed variables; factors are hypothesised causes of them, and the arrows point the other way. Principal components always exist and are unique; a factor model with a given \(m\) may fit no covariance matrix at all, so factor analysis can fail in a way that principal component analysis cannot. And principal components account for total variance, including each variable’s specific noise, while factor analysis explicitly sets that noise aside in \(\Psi \) and models only what is shared.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.