Partial Least Squares Regression

Partial least squares regression, often written PLSR and also called PLS, is a two-block latent-variable method that relates a predictor matrix $X$ to a response matrix $Y$ (Geladi and Kowalski 1986; Wold, Sjostrom and Eriksson 2001). The method is used throughout chemometrics when many, collinear predictors must be connected to one or more responses.

Unlike Principal Component Analysis, which describes variation in $X$ alone, PLS constructs latent directions using information from the relationship between the $X$ and $Y$ blocks. Unlike Principal Component Regression, it does not first compute PCA of $X$ and only afterwards regress $Y$ on those scores.

That is not the same as collapsing PLS into a supervised form of PCA. PCA and PLS have different objectives. PCA seeks high-variance directions in $X$. PLS methods construct latent directions so that projected $X$ and $Y$ quantities have strong covariance under the chosen formulation (scikit-learn cross-decomposition user guide; de Jong 1993).

Spectral and other chemometric predictor matrices often have more variables than independent samples and strong collinearity among wavelengths or channels. Direct Multiple Linear Regression then operates in a poorly conditioned original-variable space (Wold et al. 2001; Geladi and Kowalski 1986).

PCR reduces $X$ without using $Y$. A high-variance direction in $X$ may be weakly related to the response, and a lower-variance direction may be the one that matters for prediction (Haaland and Thomas 1988; scikit-learn). PLS is useful in that setting because component construction uses X-Y information. That does not mean PLS always needs fewer components than PCR, or that it always predicts better.

Write $X\in {\mathbb{R}}^{n\times p}$ and $Y\in {\mathbb{R}}^{n\times q}$. PLS1 is the single-response case $q=1$. PLS2 models $q>1$ responses jointly. Separate PLS1 fits are not the same model as one joint PLS2 fit; the latent structure can differ (scikit-learn).

Geladi and Kowalski formulate the tutorial around mean-centered blocks. Scaling is a separate preprocessing choice. It depends on the variables and the application. See Normalization & Scaling. PLS does not always require autoscaling, and it should not be described as a method that must never be scaled.

For $A$ retained latent variables the two-block representation of the centered blocks is

$X_0\approx T{P}^{\mathsf{T}}+E$

$Y_0\approx U{Q}^{\mathsf{T}}+F$

with X scores $T$, Y scores $U$, X loadings $P$, Y loadings $Q$, and residuals $E$ and $F$. Wold, Sjostrom and Eriksson describe PLSR as relating X and Y while also modelling the structure of both blocks. The scikit-learn cross-decomposition user guide writes the corresponding rank-one reconstruction after sequential extraction. X weights $W$ define directions used to construct scores. Loadings describe how the blocks relate to those scores in the decomposition. Weights, scores, and loadings are different quantities. They are not interchangeable with regression coefficients.

For a current centered residual block, a score may be written $t_a=X_a{w}_a$. After sequential extraction this is not automatically $T=XW$ for the original $X$ without a projection or rotation matrix. Software stores that mapping separately.

PLSR is the modeling method. NIPALS and SIMPLS are algorithms used to compute PLS models. They are not synonyms for PLS. NIPALS is documented on the NIPALS page. de Jong introduced SIMPLS as an alternative that forms factors as linear combinations of the original variables and avoids the explicit deflation of both data matrices used in NIPALS-style formulations (de Jong 1993). MATLAB plsregress uses SIMPLS (MathWorks). scikit-learn PLSRegression is documented as a NIPALS-related regression-mode estimator, not as SIMPLS (scikit-learn).

The integer $A$ is a hyperparameter. Do not choose it from training fit alone, from a universal X-variance percentage, or from the number of analytes. For prediction, evaluate $A$ by validation with preprocessing and PLS fitted inside each training fold. Do not use a held-out test set to choose $A$.

The identities below use two-block notation for the centered blocks. Sequential deflation is the classical NIPALS-style description. It is not the SIMPLS algorithm used by MATLAB plsregress. Covariance objectives are formulation-dependent. They are not a single universal definition of every PLS variant.

Same UCI Wine calibration as the PCR page: alcohol from the other 12 variables, same five-fold splits, StandardScaler inside each training fold. PLS uses scale=False after that scaler. The starred RMSECV is the lowest mean of fold RMSEs on this single CV path, matching cross_val_score. It is not the pooled RMSECV of the Cross-Validation page, and it is not a nested estimate of a component-selection rule. Lowest PLS RMSECV is 0.561 at $A=3$. PCR is an overlay under the same protocol, not a claim that one method always wins.

The public Python implementation is scikit-learn PLSRegression. The estimator default is scale=True, which centers and scales both blocks. Set scale=False for centering only. Compare PCR and PLS only under a stated shared preprocessing convention. Do not let software defaults decide the experiment silently.

x_weights_, x_loadings_, and x_rotations_ are different matrices. Weights are the left singular vectors of the cross-covariance matrices at each iteration. Loadings belong to the X decomposition. Rotations are the projection matrix used by transform. Prediction uses predict. Single-response $y$ may return a 1D array. Multivariate $Y$ keeps all $q$ columns. Do not flatten a PLS2 prediction to one column.

Cross validation calls cross_val_score on the estimator so that scaling and PLS are refit inside each training fold. Do not transform the full matrix first and then cross validate a regressor on those scores.

MATLAB uses official plsregress, which implements SIMPLS. With the default intercept, MathWorks centers $X$ and $Y$ and does not rescale columns. If standardized variables are required, scale them explicitly, for example with zscore, using training parameters on new data. Prediction with intercept is $\hat{Y}=[1X]\mathrm{BETA}$. Intermediate weights and loadings from MATLAB and scikit-learn should not be expected to match under every configuration.

This page does not publish a handwritten NIPALS implementation. That belongs on the NIPALS resource.

import numpy as npfrom sklearn.cross_decomposition import PLSRegressionfrom sklearn.model_selection import KFold, cross_val_score  def make_pls(n_components, scale=True):    n_components = int(n_components)    if n_components < 1:        raise ValueError("n_components must be a positive integer.")    return PLSRegression(n_components=n_components, scale=scale)  def _validate_xy(X, Y):    X = np.asarray(X, dtype=float)    Y = np.asarray(Y, dtype=float)    if X.ndim != 2:        raise ValueError("X must be a 2D array of samples by variables.")    if Y.ndim == 1:        Y_work = Y.reshape(-1, 1)        one_d = True    elif Y.ndim == 2:        Y_work = Y        one_d = False    else:        raise ValueError("Y must be a 1D vector or a 2D array of samples by responses.")    if X.shape[0] != Y_work.shape[0]:        raise ValueError("X and Y must have the same number of observations.")    if not np.all(np.isfinite(X)) or not np.all(np.isfinite(Y_work)):        raise ValueError("X and Y must contain only finite values.")    return X, Y_work, one_d  def fit_pls(X, Y, n_components, scale=True):    X, Y_work, one_d = _validate_xy(X, Y)    if n_components > min(X.shape):        raise ValueError("n_components cannot exceed min(n_samples, n_features).")    y_fit = Y_work.ravel() if one_d else Y_work    model = make_pls(n_components, scale=scale)    model.fit(X, y_fit)    return model  def predict_pls(model, X_new):    X_new = np.asarray(X_new, dtype=float)    if X_new.ndim != 2:        raise ValueError("X_new must be a 2D array of samples by variables.")    if not np.all(np.isfinite(X_new)):        raise ValueError("X_new must contain only finite values.")    return np.asarray(model.predict(X_new), dtype=float)  def pls_cv_rmse(X, y, n_components, scale=True, cv=5, random_state=0):    X = np.asarray(X, dtype=float)    y = np.asarray(y, dtype=float).reshape(-1)    if X.ndim != 2:        raise ValueError("X must be a 2D array of samples by variables.")    if X.shape[0] != y.shape[0]:        raise ValueError("X and y must have the same number of observations.")    if not np.all(np.isfinite(X)) or not np.all(np.isfinite(y)):        raise ValueError("X and y must contain only finite values.")    model = make_pls(n_components, scale=scale)    splits = KFold(n_splits=cv, shuffle=True, random_state=random_state)    scores = cross_val_score(        model,        X,        y,        cv=splits,        scoring="neg_root_mean_squared_error",    )    return float(-np.mean(scores)) 

1. 1.

   Geladi, P., & Kowalski, B. R. (1986). Partial least-squares regression: a tutorial. Analytica Chimica Acta, 185, 1-17.

   doi:10.1016/0003-2670(86)80028-9
2. 2.

   Haaland, D. M., & Thomas, E. V. (1988). Partial least-squares methods for spectral analyses. 1. Relation to other quantitative calibration methods and the extraction of qualitative information. Analytical Chemistry, 60(11), 1193-1202.

   doi:10.1021/ac00162a020
3. 3.

   Wold, S., Sjostrom, M., & Eriksson, L. (2001). PLS-regression: a basic tool of chemometrics. Chemometrics and Intelligent Laboratory Systems, 58(2), 109-130.

   doi:10.1016/S0169-7439(01)00155-1
4. 4.

   de Jong, S. (1993). SIMPLS: an alternative approach to partial least squares regression. Chemometrics and Intelligent Laboratory Systems, 18(3), 251-263.

   doi:10.1016/0169-7439(93)85002-X
5. 5.

   scikit-learn Developers (n.d.). PLSRegression and Cross decomposition. scikit-learn documentation.

6. 6.

   The MathWorks, Inc. (n.d.). plsregress. MATLAB documentation.