Open Lab/Regression & Calibration · Method

Principal Component Regression

PCR

A calibration method that performs PCA on the predictors without using the response, then regresses the response on a selected set of principal component scores.

RegressionCalibrationPCALatent VariablesDimensionality ReductionSpectroscopyChemometricsPythonMATLAB

X

PCA

T_{A}

OLS

y^

What is PCR?

Principal component regression is a multivariate calibration method that combines Principal Component Analysis with least squares regression. Keithley, Wightman and Heien describe PCR as PCA followed by a regression that relates the response to distances along the retained principal components.

They also discuss residual analysis as a quality-control companion to PCR concentration determination.

The predictor matrix is not used in its original columns as in Multiple Linear Regression. PCR first represents centered $X$ by principal component scores, then regresses the response on a selected subset of those scores.

With the notation of the PCA page, a rank $A$ approximation is

$X_{c} \approx T_{A} P_{A}^{T}$

with scores $T_{A} = X_{c} P_{A}$ . PCR then uses

$y_{c} \approx T_{A} c$

in score space. The PCA stage is constructed from $X$ without using $y$ . That distinction remains throughout this page.

Why use PCR?

Spectral predictor matrices often have many variables, strong collinearity, and more wavelengths than independent calibration directions. Direct MLR on those columns can be poorly conditioned. PCR replaces the original predictors with orthogonal PCA scores and performs the regression in a reduced component space (Haaland and Thomas 1988; MathWorks).

MathWorks documents PCR and PLSR as methods for a response when predictors are numerous and highly correlated or collinear. PCR creates components to explain variability in the predictors without using the response. That can be useful. It does not mean that PCR always predicts better than MLR, that it automatically removes noise, or that it always needs fewer components than PLS.

How does PCR work?

Training estimates the column means of $X$ and the mean of $y$ , centers both, computes PCA on the centered predictors, retains the first $A$ loadings, and fits OLS of $y_{c}$ on $T_{A}$ . New observations are centered with the training $μ_{X}$ , projected with the training $P_{A}$ , and predicted without refitting PCA.

PCA is unsupervised: it does not require $y$ . PCR as a complete calibration is supervised because $y$ is used in the regression stage. The component construction step itself remains unsupervised with respect to $y$ .

The integer $A$ is a model hyperparameter. Explained variance in $X$ and predictive relevance for $y$ are not the same quantity. Jolliffe showed that principal components associated with relatively small variance in $X$ can still be important for regression. That is not a claim that low variance components are always important. It is a warning against treating X variance order as a universal selection rule for predicting $y$ .

For predictive PCR, $A$ should be evaluated against a prediction criterion, typically by validation or cross validation. Haaland and Thomas discuss selecting loading vectors to optimize calibration models while reducing overfitting. Do not fit PCA on the complete dataset and then cross validate only the regression. PCA is a learned transformation, so it belongs inside each training fold. A scikit-learn Pipeline does that by construction.

PLS regression, treated on the PLSR page, constructs latent directions using information from both predictors and the response (Haaland and Thomas 1988; MathWorks). PCR does not. MathWorks notes that PLSR therefore often fits $y$ with fewer components than PCR. Whether that is a more parsimonious model in practice depends on the context. This page does not claim that PLS is universally superior.

Mathematics & algorithm

The identities below use the centered formulation for a model with an intercept. They reuse the PCA page for $X_{c}$ , $T_{A}$ , and $P_{A}$ , and the MLR page for OLS in score space. If an intercept is omitted, do not apply the centering of $y$ shown here.

Centering X_{c} = X - 1 μ_{X}^{T} (1) y_{c} = y - y ˉ 1 (2) μ_{X} training column means of X y ˉ training mean of y Interpretation Training means are stored and reused for new observations. They are not recomputed from test rows.

PCA scores and score-space regression X_{c} \approx T_{A} P_{A}^{T} (3) T_{A} = X_{c} P_{A} (4) y_{c} = T_{A} c_{A} + e (5) c^_{A} = argmin_{c} ∥ y_{c} - T_{A} c ∥_{2}^{2} (6) c^_{A} = (T_{A}^{T} T_{A})^{- 1} T_{A}^{T} y_{c} (7) T_{A} \in R^{n \times A} training scores for A retained components P_{A} \in R^{p \times A} loadings for A retained components c_{A} regression coefficients in score space Interpretation Equation 7 holds when T_{A} has full column rank. Score columns from distinct retained PCA directions are orthogonal under the standard construction. Equation 7 is a mathematical expression, not the recommended numerical solver.

Predictor-space coefficients and prediction y^_{c} = T_{A} c^_{A} = X_{c} P_{A} c^_{A} (8) β^_{PCR} = P_{A} c^_{A} (9) y^= y ˉ + X_{c} β^_{PCR} (10) β^_{0} = y ˉ - μ_{X}^{T} β^_{PCR} (11) y^= β^_{0} + X β^_{PCR} (12) Interpretation Equations 10 and 12 are the same centered-training predictor written in centered X and in original X . Do not mix the two conventions. For new data, center with the training mean: y^_{new} = y ˉ + T_{new} c^_{A} with T_{new} = (X_{new} - 1 μ_{X}^{T}) P_{A} .

SVD form X_{c} = U Σ V^{T} (13) P_{A} = V_{A}, T_{A} = U_{A} Σ_{A} (14) Interpretation This is the conversion already used on the PCA page. PCR then regresses y_{c} on U_{A} Σ_{A} . SVD itself is the factorization, not the calibration method.

If all principal components that span the column space of $X_{c}$ are retained, and that matrix has full column rank, PCR represents the same least squares predictor space as centered OLS. Truncation to $A$ smaller than that rank is a restriction, not an identity.

Algorithm 1

Principal Component Regression

Inputtraining predictor matrix X, training response y, number of retained components A

OutputPCA model, PCR coefficients, predictions

01 $μ_{X} \leftarrow column means of training X$
02 $y ˉ \leftarrow mean of training y$
03 $X_{c} \leftarrow X - 1 μ_{X}^{T}$
04 $y_{c} \leftarrow y - y ˉ 1$
05perform PCA on centered training X
06 $P_{A} \leftarrow first A loading vectors$
07 $T_{A} \leftarrow X_{c} P_{A}$
08 $c^_{A} \leftarrow argmin_{c} ∥ y_{c} - T_{A} c ∥_{2}^{2}$
09 $β^_{PCR} \leftarrow P_{A} c^_{A}$
10 $X_{new, c} \leftarrow X_{new} - μ_{X}$
11 $T_{new} \leftarrow X_{new, c} P_{A}$
12 $y^_{new} \leftarrow y ˉ + T_{new} c^_{A}$
13return PCA model, PCR coefficients, and predictions

This is a SPARKS representation of the centered PCR workflow. Step 05 delegates PCA to a validated implementation. Step 08 delegates OLS to a validated least squares solver. PCA is not refit on new observations. Preprocessing parameters are not computed from test data.

Visual example

UCI Wine: alcohol is predicted from the other 12 chemical variables. Five-fold CV fits StandardScaler, PCA, and OLS inside each training fold. The starred RMSECV is the lowest mean of fold RMSEs on this single CV path, matching cross_val_score. It is not the pooled RMSECV of the Cross-Validation page, and it is not a nested estimate of a component-selection rule. Lowest RMSECV is 0.557 at $A = 7$ . Predicted values are out-of-fold at that A.

UCI Wine PCR. Left: RMSECV versus retained components. Right: out-of-fold predicted versus measured alcohol at the CV-selected A. Aeberhard and Forina 1991, DOI 10.24432/C5PC7J.

Code

The public Python implementation is a scikit-learn Pipeline of PCA and LinearRegression. PCA centers features and does not scale them. LinearRegression fits an intercept by default. StandardScaler is optional. It centers and scales. Scaling is a preprocessing choice, not a requirement of PCR. See Normalization & Scaling. If both scaler and PCA are used, PCA receives already centered and scaled columns.

Cross validation calls cross_val_score on the pipeline so that scaling, PCA, and regression are refit inside each training fold. Do not transform the full matrix with PCA and then cross validate a regressor on those scores.

pcr_svd is a transparent centered implementation using SVD and numpy.linalg.lstsq. It matches the unscaled pipeline when the same $A$ is used. It does not apply StandardScaler.

MATLAB follows the official MathWorks PCR pattern: pca on $X$ , then least squares of centered $y$ on the selected scores, then $β^_{PCR} = P_{A} c^_{A}$ . Prediction centers new rows with the training mean stored by pca. There is no documented MATLAB function named pcr in this workflow.

pcr

import numpy as npfrom sklearn.decomposition import PCAfrom sklearn.linear_model import LinearRegressionfrom sklearn.model_selection import KFold, cross_val_scorefrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScaler  def make_pcr_pipeline(n_components, scale=False):    n_components = int(n_components)    if n_components < 1:        raise ValueError("n_components must be a positive integer.")     steps = []    if scale:        steps.append(("scale", StandardScaler()))    steps.append(("pca", PCA(n_components=n_components)))    steps.append(("regression", LinearRegression()))    return Pipeline(steps)  def fit_pcr(X, y, n_components, scale=False):    X = np.asarray(X, dtype=float)    y = np.asarray(y, dtype=float).reshape(-1)    if X.ndim != 2:        raise ValueError("X must be a 2D array of samples by variables.")    if X.shape[0] != y.shape[0]:        raise ValueError("X and y must have the same number of observations.")    if not np.all(np.isfinite(X)) or not np.all(np.isfinite(y)):        raise ValueError("X and y must contain only finite values.")    if n_components > min(X.shape):        raise ValueError("n_components cannot exceed min(n_samples, n_features).")     model = make_pcr_pipeline(n_components, scale=scale)    model.fit(X, y)    return model  def predict_pcr(model, X_new):    X_new = np.asarray(X_new, dtype=float)    if X_new.ndim != 2:        raise ValueError("X_new must be a 2D array of samples by variables.")    if not np.all(np.isfinite(X_new)):        raise ValueError("X_new must contain only finite values.")    return np.asarray(model.predict(X_new), dtype=float)  def pcr_cv_rmse(X, y, n_components, scale=False, cv=5, random_state=0):    X = np.asarray(X, dtype=float)    y = np.asarray(y, dtype=float).reshape(-1)    pipeline = make_pcr_pipeline(n_components, scale=scale)    splits = KFold(n_splits=cv, shuffle=True, random_state=random_state)    scores = cross_val_score(        pipeline,        X,        y,        cv=splits,        scoring="neg_root_mean_squared_error",    )    return float(-np.mean(scores))  def pcr_svd(X, y, n_components):    X = np.asarray(X, dtype=float)    y = np.asarray(y, dtype=float).reshape(-1)    n_components = int(n_components)    if X.ndim != 2:        raise ValueError("X must be a 2D array of samples by variables.")    if X.shape[0] != y.shape[0]:        raise ValueError("X and y must have the same number of observations.")    if not np.all(np.isfinite(X)) or not np.all(np.isfinite(y)):        raise ValueError("X and y must contain only finite values.")    if n_components < 1:        raise ValueError("n_components must be a positive integer.")     X_mean = X.mean(axis=0)    y_mean = float(y.mean())    Xc = X - X_mean    yc = y - y_mean    _U, _s, Vt = np.linalg.svd(Xc, full_matrices=False)    if n_components > Vt.shape[0]:        raise ValueError("n_components cannot exceed the SVD rank.")     P = Vt.T    P_A = P[:, :n_components]    T_A = Xc @ P_A    c_hat, *_ = np.linalg.lstsq(T_A, yc, rcond=None)    beta_pcr = P_A @ c_hat    intercept = y_mean - X_mean @ beta_pcr    return {        "mean_X": X_mean,        "mean_y": y_mean,        "loadings": P_A,        "scores": T_A,        "c_hat": c_hat,        "beta": beta_pcr,        "intercept": intercept,    }  def predict_pcr_svd(model, X_new):    X_new = np.asarray(X_new, dtype=float)    Xc_new = X_new - model["mean_X"]    return model["mean_y"] + Xc_new @ model["beta"]

Practical notes

PCR combines PCA with regression. The PCA stage is constructed from X without using y.
The complete PCR calibration is supervised because y is used in the regression stage.
The retained component number A is a model hyperparameter, not a fixed property of X.
Explained X variance alone should not be treated as a universal criterion for selecting PCR components.
Low variance principal components can sometimes be relevant to predicting y. They are not always relevant, and they are not always irrelevant.
Cross validation should fit preprocessing, PCA, and regression inside each training fold.
Scaling is a preprocessing choice, not a universal PCR requirement.
PCR can help when predictors are strongly correlated or high dimensional. It does not always outperform MLR.
PCR coefficients can be mapped back to the original predictor space as $P_{A}$ times the score-space coefficients.
PCR components are not automatically chemically meaningful latent variables.

References

1.
Keithley, R. B., Wightman, R. M., & Heien, M. L. (2009). Multivariate concentration determination using principal component regression with residual analysis. TrAC Trends in Analytical Chemistry, 28(9), 1127-1136.
doi:10.1016/j.trac.2009.07.002
2.
Jolliffe, I. T. (1982). A note on the use of principal components in regression. Journal of the Royal Statistical Society. Series C (Applied Statistics), 31(3), 300-303.
doi:10.2307/2348005
3.
Haaland, D. M., & Thomas, E. V. (1988). Partial least-squares methods for spectral analyses. 1. Relation to other quantitative calibration methods and the extraction of qualitative information. Analytical Chemistry, 60(11), 1193-1202.
doi:10.1021/ac00162a020
4.
scikit-learn Developers (n.d.). Pipeline, PCA, LinearRegression, StandardScaler, and cross_val_score. scikit-learn documentation.
5.
The MathWorks, Inc. (n.d.). pca. MATLAB documentation.
6.
The MathWorks, Inc. (n.d.). Partial least squares regression and principal components regression. MATLAB documentation.

← Back to Open Lab