14.4 Factor Extraction Methods


  1. principal components
  2. Iterated components
  3. Maximum Likelihood
  • Choose the first \(m\) principal components and modify them to fit the factor model defined in the previous section.
  • They explain the greatest proportion of the variance and are therefore the most important

14.4.1 Principal components (PC Factor model)

Recall that \(\mathbf{C} = \mathbf{A}\mathbf{X}\), C’s are a function of X

\[ C_{1} = a_{11}X_{1} + a_{12}X_{2} + \ldots + a_{1P}X_{p} \]

We want the reverse: X’s are a function of F’s.

  • Use the inverse! –> If \(c = 5x\) then \(x = 5^{-1}C\)

The inverse PC model is \(\mathbf{X} = \mathbf{A}^{-1}\mathbf{C}\).

Since \(\mathbf{A}\) is orthogonal, \(\mathbf{A}^{-1} = \mathbf{A}^{T} = \mathbf{A}^{'}\), so

\[ X_{1} = a_{11}C_{1} + a_{21}C_{2} + \ldots + a_{P1}C_{p} \]

But there are more PC’s than Factors…

\[ \begin{equation} \begin{aligned} X_{i} &= \sum_{j=1}^{P}a_{ji}C_{j} \\ &= \sum_{j=1}^{m}a_{ji}C_{j} + \sum_{j=m+1}^{m}a_{ji}C_{j} \\ &= \sum_{j=1}^{m}l_{ji}F_{j} + e_{i} \\ \end{aligned} \end{equation} \]


  • \(V(C_{j}) = \lambda_{j}\) not 1
  • We transform: \(F_{j} = C_{j}\lambda_{j}^{-1/2}\)
  • Now \(V(F_{j}) = 1\)
  • Loadings: \(l_{ij} = \lambda_{j}^{1/2}a_{ji}\)
\(l_{ij}\) is the correlation coefficient between variable \(i\) and factor \(j\)

This is similar to \(a_{ij}\) in PCA.

14.4.2 Iterated components

Select common factors to maximize the total communality

  1. Get initial communality estimates
  2. Use these (instead of original variances) to get the PC’s and factor loadings
  3. Get new communality estimates
  4. Rinse and repeat
  5. Stop when no appreciable changes occur. R code

Not shown, but can be obtained using the factanal package in R.

14.4.3 Maximum Likelihood

  • Assume that all the variables are normally distributed
  • Use Maximum Likelihood to estimate the parameters

14.4.4 R code

The cutoff argument hides loadings under that value for ease of interpretation. Here I am setting that cutoff at 0 so that all loadings are being displayed.

The uniqueness’s (\(u^{2}\)) for X2, X4, X5 are pretty low. The factor equations now are:

\[ \begin{equation} \begin{aligned} X_{1} &= -0.06F_{1} + 0.79F_{2} + e_{1} \\ X_{2} &= -0.07F_{1} + 1F_{2} + e_{2} \\ X_{3} &= 0.58F_{1} + 0.19F_{2} + e_{3} \\ \vdots \end{aligned} \end{equation} \]

Notice that neither extraction method reproduced our true hypothetical factor model. Rotating the factors will achieve our desired results.