Open Lab/Exploratory Analysis · Method

Principal Component Analysis

PCA

A multivariate method that represents correlated data using orthogonal directions that capture decreasing amounts of variance.

ChemometricsMultivariate AnalysisDimensionality ReductionExploratory AnalysisScoresLoadingsSVDPythonMATLAB

What is PCA?

Principal Component Analysis is a multivariate method that constructs orthogonal directions called principal components. The first principal component identifies the direction along which the centered observations show the greatest variance. Each subsequent component is orthogonal to the preceding components and captures the greatest remaining variance under that orthogonality constraint.

PCA represents observations using coordinates called scores. The coefficients that define the component directions in the original variable space are commonly called loadings. The method is unsupervised: it does not use a response variable, and it does not automatically discover chemically meaningful variables.

Wold, Esbensen and Geladi present this bilinear scores and loadings framework as a chemometric tool for data exploration. Bro and Smilde give a modern geometric and interpretational account of the same construction. Pearson formulated the related geometry of closest fit in 1901.

Why use PCA?

High dimensional measurements are often strongly correlated. PCA provides a smaller set of orthogonal coordinates that represent the dominant systematic variation in the centered data. That reduced representation is used to explore patterns, not to declare what those patterns mean chemically.

Typical uses supported by the chemometric literature include inspecting dominant systematic structure, representing many variables with fewer coordinates, visualizing relationships among observations, studying relationships among variables, flagging unusual observations or patterns for further investigation, and reducing dimensionality while retaining a chosen portion of variance.

Explained variance is variance. It is not automatically information, chemical content, or class structure. Low variance is not automatically noise. PCA does not discover causal relations, and it does not need to precede every subsequent model.

How does PCA work?

Start from a data matrix with correlated variables. Subtract the variable means so that the origin sits at the center of the cloud. Find the unit direction of maximum variance in that centered space, and project the observations onto it. Then find the next orthogonal direction that captures the largest remaining variance. The projections are scores. The direction coefficients are loadings.

Conventional covariance based PCA is applied to centered data. Centering is part of that default formulation. Scaling is not. Autoscaling changes the geometry and the relative influence of variables, so PCA after autoscaling is a different analysis. scikit-learn PCA centers each feature and does not scale features automatically. Those two choices are documented separately from the decomposition itself. Scaling decisions belong with Normalization & Scaling.

Each row of the score matrix corresponds to one observation. A score plot therefore shows samples in the reduced component coordinate system. Each column of the loading matrix is one principal direction in the original variable space. A large absolute loading indicates a strong contribution to that component direction under the chosen scaling. It does not by itself establish chemical importance or causality.

The sign of a principal component is not unique. If $p_{k}$ is a valid loading vector, then $- p_{k}$ represents the same axis, and the corresponding score vector changes sign. Two correct implementations may therefore return reversed signs while representing the same model. A sign flip by itself is not disagreement.

Mathematics & algorithm

Let $X \in R^{n \times p}$ have observations as rows and variables as columns. The sample covariance convention below uses denominator $n - 1$ .

Centering μ = \frac{1}{n} i = 1 \sum n x_{i} (1) X_{c} = X - 1 μ^{T} (2) n number of observations p number of variables x_{i} row i of X μ vector of variable means X_{c} column-centered data matrix 1 length-n vector of ones Interpretation Equation 2 places the origin at the variable means. This is the default input to covariance based PCA. Scaling each column is a separate modelling choice and is not implied by these two equations.

First principal component ∥ p_{1} ∥_{2} = 1 (3) t_{1} = X_{c} p_{1} (4) p_{1} = argmax_{∥ p ∥_{2} = 1} Var (X_{c} p) (5) p_{1} first loading vector, a unit direction in variable space t_{1} first score vector, one coordinate per observation Interpretation The first component is the unit direction of greatest variance in the centered data. Later components repeat the same criterion in the subspace orthogonal to the directions already found.

Sample covariance eigenproblem S = \frac{1}{n - 1} X_{c}^{T} X_{c} (6) S p_{k} = λ_{k} p_{k} (7) λ_{1} \geq λ_{2} \geq \dots \geq 0 (8) S sample covariance matrix of the columns of X p_{k} loading direction of component k λ_{k} variance of the scores on component k under this covariance convention Interpretation The eigenvectors of S are the loading directions. The corresponding eigenvalues are the score variances. The ordering in Equation 8 is part of the definition of successive principal components.

PCA matrix model X_{c} = T P^{T} + E (9) X_{c} \approx T_{A} P_{A}^{T} (10) T_{A} = X_{c} P_{A} (11) X^_{c} = T_{A} P_{A}^{T} (12) X^= X^_{c} + 1 μ^{T} (13) E = X_{c} - T_{A} P_{A}^{T} (14) T_{A} \in R^{n \times A} scores for A retained components P_{A} \in R^{p \times A} loadings for A retained components X^_{c} reconstruction of the centered data X^reconstruction in the original coordinates E residual matrix after A components Interpretation Equation 11 holds when the loading columns are orthonormal. Residuals are the part of the centered matrix not represented by the retained components. They are not automatically noise. A full-rank model with all available components reconstructs X_{c} within numerical precision.

Explained variance R_{k} = \frac{λ _{k}}{\sum _{j} λ _{j}} (15) R_{cum, A} = k = 1 \sum A R_{k} (16) R_{k} explained variance ratio of component k R_{cum, A} cumulative explained variance after A components Interpretation These ratios describe variance represented by the components. They are not a percentage of chemical information. A threshold such as 95 percent may be used as an example cutoff in a specific study. It is not a universal rule for choosing A.

Computation via SVD X_{c} = U Σ V^{T} (17) P = V, T = U Σ = X_{c} V (18) λ_{k} = \frac{σ _{k}^{2}}{n - 1} (19) U left singular vectors of the centered matrix Σ diagonal matrix of singular values V right singular vectors, used here as loadings σ_{k} singular value k Interpretation SVD is introduced here only as a numerically robust way to obtain the PCA factors. scikit-learn computes PCA using SVD based methods on centered input. The conversion in Equation 19 is the sample covariance convention with denominator n minus 1.

Algorithm 1

Principal Component Analysis using SVD

InputData matrix X ∈ ℝⁿˣᵖ, number of retained components A

OutputMean vector $μ$ , scores $T_{A}$ , loadings $P_{A}$ , explained variances $λ_{A}$

01 $μ \leftarrow column means of X$
02 $X_{c} \leftarrow X - 1 μ^{T}$
03 $[U, Σ, V] \leftarrow SVD (X_{c})$
04 $P \leftarrow V$
05 $T \leftarrow U Σ$
06 $λ_{k} \leftarrow σ_{k}^{2} / (n - 1)$
07 $order components by decreasing λ_{k}$
08 $retain the first A components$
09 $P_{A} \leftarrow P_{[:, 1 : A]}$
10 $T_{A} \leftarrow T_{[:, 1 : A]}$
11 $λ_{A} \leftarrow λ_{1 : A}$
12return $μ, T_{A}, P_{A}, λ_{A}$

This pseudocode is a SPARKS representation of Equations 1 to 19. It is not a verbatim extract from the cited papers. The third SVD factor is $V$ , the right singular vectors, matching MATLAB svd and the SVD page. NumPy np.linalg.svd returns $V^{T}$ as its third factor. SVD of a centered matrix already returns singular values in decreasing order. The explicit ordering step is retained for clarity. New observations are scored with the training mean and the retained loadings, not by refitting PCA.

Visual example

UCI Wine: 178 samples and 13 chemical variables, autoscaled, then unsupervised PCA. Cultivar color is applied after the decomposition. Autoscaling is a choice for that visual only. It is not the scikit-learn PCA default. PC1 and PC2 account for 36.2% and 19.2% of the sample variance in this matrix.

UCI Wine after autoscaling. Left: scores. Right: variable loadings. Cultivar color is post-hoc. Aeberhard and Forina 1991, DOI 10.24432/C5PC7J.

Code

The Python functions wrap the official scikit-learn PCA estimator. Rows of X are observations and columns are variables. scikit-learn centers the variables and does not scale them. The array components_ has shape n_components by n_features, so components_.T is the loadings matrix with variables as rows and components as columns, matching $P$ on this page.

After fitting, new observations are projected with transform_pca, which calls the fitted estimator. That uses the training mean and the learned loadings. Do not fit a new PCA independently on test data when the decomposition is part of a predictive workflow.

pca_svd is a transparent SVD verification of the same model. It is not a replacement for the public scikit-learn API. Component-wise comparison of scores and loadings must allow sign flips.

The MATLAB functions use the MathWorks pca interface when it is available. coeff contains loading coefficients, score contains observation scores, latent contains principal component variances, explained contains the percentage of total variance, and mu contains the variable means. New observations are scored as $(X_{new} - μ) P_{A}$ . pcaBySvd is an educational SVD path that uses only base MATLAB.

import numpy as npfrom sklearn.decomposition import PCA  def fit_pca(X, n_components=None):    X = np.asarray(X, dtype=float)     if X.ndim != 2:        raise ValueError(            "X must be a 2D array of samples by variables."        )     if not np.all(np.isfinite(X)):        raise ValueError(            "X must contain only finite values."        )     if X.shape[0] < 2:        raise ValueError(            "PCA requires at least two observations."        )     pca = PCA(n_components=n_components)    scores = pca.fit_transform(X)    loadings = pca.components_.T     return {        "pca": pca,        "mean": pca.mean_,        "scores": scores,        "loadings": loadings,        "explained_variance": pca.explained_variance_,        "explained_ratio": pca.explained_variance_ratio_,    }  def transform_pca(X_new, model):    X_new = np.asarray(X_new, dtype=float)     if X_new.ndim != 2:        raise ValueError(            "X_new must be a 2D array of samples by variables."        )     if not np.all(np.isfinite(X_new)):        raise ValueError(            "X_new must contain only finite values."        )     return model["pca"].transform(X_new)  def pca_svd(X, n_components=None):    X = np.asarray(X, dtype=float)    n = X.shape[0]    mean = X.mean(axis=0)    X_centered = X - mean    U, singular_values, Vt = np.linalg.svd(        X_centered,        full_matrices=False,    )    explained_variance_full = singular_values**2 / (n - 1)    total = explained_variance_full.sum()     if n_components is not None:        U = U[:, :n_components]        singular_values = singular_values[:n_components]        Vt = Vt[:n_components]        explained_variance_full = explained_variance_full[:n_components]     scores_svd = U * singular_values    loadings_svd = Vt.T    explained_variance_svd = explained_variance_full     return {        "mean": mean,        "centered": X_centered,        "scores": scores_svd,        "loadings": loadings_svd,        "explained_variance": explained_variance_svd,        "explained_ratio": explained_variance_svd / total,        "singular_values": singular_values,    }

Practical notes

PCA is unsupervised and does not use a response variable.
Centering is normally part of conventional covariance based PCA.
Scaling is a separate choice. PCA on centered variables is not the same analysis as PCA after autoscaling. See Normalization & Scaling.
Scores describe observations in component space. Loadings describe component directions in the original variable space.
A large absolute loading should be interpreted in context. It is not automatically an indicator of causal or chemical importance.
The sign of scores and loadings can flip without changing the PCA model.
Explained variance is variance. It is not necessarily useful chemical information, and low variance is not automatically noise.
The number of retained components should be chosen according to the analytical purpose. A universal explained variance threshold is not a scientific rule.
When PCA is used before a predictive model, transform new observations with the training mean and the training loadings. Do not refit PCA on the test matrix.

References

1.
Wold, S., Esbensen, K., & Geladi, P. (1987). Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3), 37-52.
doi:10.1016/0169-7439(87)80084-9
2.
Bro, R., & Smilde, A. K. (2014). Principal component analysis. Analytical Methods, 6(9), 2812-2831.
doi:10.1039/C3AY41907J
3.
Bro, R., & Smilde, A. K. (2003). Centering and scaling in component analysis. Journal of Chemometrics, 17(1), 16-33.
doi:10.1002/cem.773
4.
Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11), 559-572.
doi:10.1080/14786440109462720
5.
scikit-learn Developers (n.d.). sklearn.decomposition.PCA. scikit-learn documentation.
6.
The MathWorks, Inc. (n.d.). pca. MATLAB documentation.
7.
Aeberhard, S., & Forina, M. (1991). Wine. UCI Machine Learning Repository. Used only for the Signature Visual.
doi:10.24432/C5PC7J

← Back to Open Lab