Open Lab/Model Evaluation · Concept

Cross-Validation

A resampling procedure that repeatedly fits a model on one subset of observations and evaluates it on a held-out subset, used for model evaluation and model selection under a stated split design.

ChemometricsCalibrationRegressionSupervised LearningPythonMATLAB

val / train → RMSECV

What is Cross-Validation?

Cross-validation is a resampling procedure. Observations are partitioned, repeatedly, into a subset used to fit a model and a held-out subset used to evaluate that fit. Predictions for a held-out observation come from a model that was not fitted on that observation.

Westad and Marini treat an independent, representative test set as the most conservative numerical validation for prediction. They describe cross-validation as generally second-best, and useful when the number of objects is limited. Cross-validation estimates predictive behavior under the chosen split design. It does not state how the model will perform on future data in general, it does not prevent overfitting, and it does not replace an independent test set.

Why use it?

Chemometric calibration often has limited $n$ relative to the number of variables, and a complexity choice such as the number of PLS latent variables or retained principal components. Holding out a large test set can leave too few objects for a stable fit. Cross-validation lets each observation serve in training in some folds and in validation in another fold.

Repeated use of the same observations does not create new independent samples. Westad and Marini also use calibration-set CV at the appropriate grouping level to choose model rank even when a test set exists, so that rank is not too optimistic before the test set is predicted.

How K-fold Cross-Validation works

Ordinary $K$ -fold CV partitions the $n$ observations into $K$ folds:

$D = D_{1} \cup D_{2} \cup \dots \cup D_{K}$ .

For fold $k$ , the training set is $D ∖ D_{k}$ and the validation set is $D_{k}$ . A new model is fitted for every fold. Validation predictions must come from that fold's training fit. In scikit-learn KFold with $K = n$ this is leave-one-out. The first $n mod K$ folds receive one extra sample when $n$ is not divisible by $K$ .

Mathematics

Samples are rows. Let $I_{k}$ be the validation index set of fold $k$ , and let $y^_{i}^{(- k)}$ be the prediction for observation $i$ from the model fitted without fold $k$ .

Notation X \in R^{n \times p}, y \in R^{n}, I_{1}, \dots, I_{K} (1) n number of observations (rows) p number of predictor variables (columns) K number of folds I_{k} validation indices held out in fold k y^_{i}^{(- k)} prediction for i from a model fitted without fold k Interpretation The folds partition the n indices. Training indices for fold k are the complement of I_{k} .

Fold mean squared error MSE_{k} = \frac{1}{∣ I _{k} ∣} i \in I_{k} \sum (y_{i} - y^_{i}^{(- k)})^{2} (2) Interpretation Each fold has its own validation size. The mean of the K fold RMSE values is therefore not guaranteed to equal the RMSE computed from all n out-of-fold residuals.

Pooled RMSECV (this page) MSE_{C V} = \frac{1}{n} k = 1 \sum K i \in I_{k} \sum (y_{i} - y^_{i}^{(- k)})^{2} (3) RMSECV = MSE_{C V} (4) Interpretation Equations 3 and 4 are the convention used by the educational code on this page. Westad and Marini report RMSECV as a figure of merit; they do not display this aggregation formula. scikit-learn cross_val_score with neg_root_mean_squared_error reports the mean of fold RMSEs. Open Lab PLSR and PCR figures used that sklearn fold-mean score. MATLAB, Python, and chemometric software can differ. State the convention before comparing numbers.

K-fold algorithm

Algorithm 1 is ordinary $K$ -fold CV with learned preprocessing inside each training fold. Sample-wise formulas that do not estimate parameters from other observations are outside this rule. Mean centering, column scaling (including sklearn StandardScaler, which uses population standard deviation rather than the sample autoscaling of the Normalization and Scaling page), PCA, feature selection, and imputation estimate parameters from a collection of rows and must be fitted on the training fold only.

Algorithm 1

K-fold cross-validation

InputX, y, K, pipeline M with learned transforms and a model, loss L

Outputout-of-fold predictions and the stated CV summary

01Require: X (n by p), y (n), K folds, modeling pipeline M, loss L
02partition indices into $I_{1}, \dots, I_{K}$
03for $k = 1, \dots, K$
04training indices $\leftarrow {1, \dots, n} ∖ I_{k}$
05validation indices $\leftarrow I_{k}$
06fit learned transformations on the training fold only
07transform training and validation using those fitted parameters
08fit the model on the transformed training fold
09predict the transformed validation fold; store $y^_{i}^{(- k)}$
10aggregate stored predictions with the stated metric
11return cross-validation result

A fresh model is fitted for every fold. Learned transforms use training-fold observations only. The aggregation step on this page is pooled RMSECV (equations 3 and 4). sklearn cross_validate instead averages fold scores. Do not preprocess the full X with a fitted transform and then cross-validate the already transformed matrix.

Choosing a validation strategy

Random $K$ -fold splitting is for observations that can reasonably be treated as exchangeable. Westad and Marini state that automatic calibration/test splitting is justified only when there is no systematic stratification of objects that can affect the result. Chemometric tables often contain replicates, batches, instruments, lots, subjects, time order, or sample families. Those structures belong in the validation design.

If several spectra are technical replicates of the same physical sample, and the intended unit of generalization is that physical sample, placing those spectra on both sides of a split makes the validation problem too easy. Westad and Marini keep replicates of the same physical sample together (their Case C) and also illustrate keeping physical samples, cultivars, or years out as groups. scikit-learn GroupKFold keeps a group out of training when it is in the test fold. Time dependence is a different design: TimeSeriesSplit trains on past folds and tests on a later fold. Stratified splitting applies to classification, not to the regression examples here.

Leave-one-out holds out one observation at a time and fits $n$ models. It is not automatically the most rigorous option, and it is not the default best choice for small chemometric sets. The scikit-learn user guide notes computational cost and often high variance, and states that many authors prefer 5- or 10-fold CV to LOO. That is a software-documented practice, not a chemometric law. Westad and Marini's oat example is more conservative when grouped by cultivar than when using LOO.

There is no universal $K$ . scikit-learn KFold defaults to 5 splits. MATLAB cvpartition defaults to 10 folds. Those are library defaults. Westad and Marini give a rule of thumb that CV is used when $n < 40$ ; that is their rule of thumb for preferring CV over a large held-out test fraction, not a rule for choosing $K$ .

If CV is used to choose the number of PLS or PCR components, variables, preprocessing, or other hyperparameters, that CV result is part of model selection. Do not report the same optimized value as an unbiased estimate of the completed selection procedure. Westad and Marini state that CV cannot decide the best model among many when variable selection is performed, and point to cross-model validation as a more conservative alternative. Anderssen, Dyrstad, Westad, and Martens describe over-optimism when optimization and evaluation reuse the same validation information. Filzmoser, Liebmann, and Varmuza describe repeated double CV as jointly optimizing complexity and estimating prediction error for new cases from the same population.

Nested (double) CV keeps those tasks apart. The inner loop selects components, variables, or hyperparameters. The outer loop assesses the whole selection procedure. Outer validation observations must not enter inner-loop selection. An independent external test set, left untouched during selection, is the other standard route. The inner listing below uses minimum pooled RMSECV only as one possible inner rule. It is not a universal component-selection criterion.

Repeated $K$ -fold CV repeats the partition with different randomization and shows sensitivity to a particular split. It does not make observations independent, and it should not be used to hide instability. shuffle=True is appropriate only when a random partition matches the intended prediction problem.

Visual example

The signature diagram is ordinary 5-fold CV: each row holds out one block (orange) and trains on the rest. Fold predictions are then aggregated. No chemical dataset is introduced for decoration. The figure below is procedural. It does not invent performance numbers.

Correct CV

1split into folds
2fit preprocessing on the training fold
3transform train and validation with those parameters
4fit the model on training data
5predict the validation fold

Incorrect CV

1fit data-dependent preprocessing on all observations
2then run cross-validation on the already transformed matrix
3validation observations have already informed the transform
4fold predictions are no longer held-out in that sense

Validation information leaked into the learned transform.

Correct K-fold CV fits learned preprocessing on the training fold only. Fitting a data-dependent transform on all observations before CV lets validation rows influence that transform.

Code

The Python listing implements Algorithm 1 for ordinary least squares with optional StandardScaler fitted inside each training fold. pooled_rmsecv is equations 3 and 4. Component comparison reuses the same folds. Nested CV selects a component count on inner folds of the outer training set only. Minimum inner RMSECV is one inner rule, not a required selection law.

The sklearn tab uses Pipeline and cross_validate. That RMSE score is the mean of fold RMSEs. cross_val_predict is for out-of-fold diagnostics. Current scikit-learn documentation states that it is not an appropriate measure of generalization error.

MATLAB kfoldCvOls uses consecutive folds that match scikit-learn KFold(shuffle=False): first $n mod K$ folds are larger by one. Column scaling uses the training-fold mean and population standard deviation (MATLAB std(X,1)), matching StandardScaler. Official cvpartition with KFold is a random nonstratified partition and defaults to 10 folds. It is not this consecutive split.

cross-validation

import numpy as npfrom sklearn.cross_decomposition import PLSRegressionfrom sklearn.linear_model import LinearRegressionfrom sklearn.model_selection import KFoldfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScaler  def _validate_xy(X, y):    X = np.asarray(X, dtype=float)    y = np.asarray(y, dtype=float).reshape(-1)    if X.ndim != 2:        raise ValueError("X must be a 2D array of samples by variables.")    if X.shape[0] != y.shape[0]:        raise ValueError("X and y must have the same number of observations.")    if min(X.shape) == 0:        raise ValueError("X must have at least one row and one column.")    if not np.all(np.isfinite(X)) or not np.all(np.isfinite(y)):        raise ValueError("X and y must contain only finite values.")    return X, y  def make_kfold(n_splits, shuffle=False, random_state=None):    n_splits = int(n_splits)    if n_splits < 2:        raise ValueError("n_splits must be at least 2.")    if shuffle and random_state is None:        raise ValueError("random_state is required when shuffle=True.")    return KFold(n_splits=n_splits, shuffle=shuffle, random_state=random_state)  def pooled_rmsecv(y_true, y_pred):    y_true = np.asarray(y_true, dtype=float).reshape(-1)    y_pred = np.asarray(y_pred, dtype=float).reshape(-1)    if y_true.shape[0] != y_pred.shape[0]:        raise ValueError("y_true and y_pred must have the same length.")    if y_true.shape[0] == 0:        raise ValueError("y_true must contain at least one observation.")    return float(np.sqrt(np.mean((y_true - y_pred) ** 2)))  def kfold_oof_ols(X, y, n_splits=5, shuffle=False, random_state=None, scale=True):    X, y = _validate_xy(X, y)    if n_splits > X.shape[0]:        raise ValueError("n_splits cannot exceed n_samples.")    splitter = make_kfold(n_splits, shuffle=shuffle, random_state=random_state)    y_oof = np.empty(y.shape[0], dtype=float)    folds = []    scaler_means = []    scaler_scales = []    fold_mse = []    for train, val in splitter.split(X):        folds.append((train.copy(), val.copy()))        X_train = X[train]        X_val = X[val]        if scale:            scaler = StandardScaler()            X_train = scaler.fit_transform(X_train)            X_val = scaler.transform(X_val)            scaler_means.append(scaler.mean_.copy())            scaler_scales.append(scaler.scale_.copy())        else:            scaler_means.append(X_train.mean(axis=0))            scaler_scales.append(np.ones(X.shape[1], dtype=float))        model = LinearRegression()        model.fit(X_train, y[train])        y_hat = np.asarray(model.predict(X_val), dtype=float).reshape(-1)        y_oof[val] = y_hat        fold_mse.append(float(np.mean((y[val] - y_hat) ** 2)))    return {        "y_oof": y_oof,        "rmsecv": pooled_rmsecv(y, y_oof),        "fold_mse": np.asarray(fold_mse, dtype=float),        "folds": folds,        "scaler_means": np.vstack(scaler_means),        "scaler_scales": np.vstack(scaler_scales),        "splitter": splitter,    }  def cv_rmsecv_by_n_components(    X, y, n_components_list, n_splits=5, shuffle=False, random_state=None):    X, y = _validate_xy(X, y)    splitter = make_kfold(n_splits, shuffle=shuffle, random_state=random_state)    folds = [(train.copy(), val.copy()) for train, val in splitter.split(X)]    n_components_list = [int(a) for a in n_components_list]    scores = []    for a in n_components_list:        if a < 1:            raise ValueError("n_components must be a positive integer.")        y_oof = np.empty(y.shape[0], dtype=float)        for train, val in folds:            pipe = Pipeline(                [                    ("scaler", StandardScaler()),                    ("pls", PLSRegression(n_components=a, scale=False)),                ]            )            pipe.fit(X[train], y[train])            y_oof[val] = np.asarray(pipe.predict(X[val]), dtype=float).reshape(-1)        scores.append(pooled_rmsecv(y, y_oof))    return {        "n_components": np.asarray(n_components_list),        "rmsecv": np.asarray(scores, dtype=float),        "folds": folds,    }  def nested_cv_select_n_components(    X,    y,    n_components_list,    n_splits_outer=5,    n_splits_inner=4,    shuffle=False,    random_state=None,):    X, y = _validate_xy(X, y)    outer = make_kfold(        n_splits_outer, shuffle=shuffle, random_state=random_state    )    y_oof = np.empty(y.shape[0], dtype=float)    records = []    for outer_train, outer_val in outer.split(X):        inner = cv_rmsecv_by_n_components(            X[outer_train],            y[outer_train],            n_components_list,            n_splits=n_splits_inner,            shuffle=shuffle,            random_state=random_state,        )        a_star = int(inner["n_components"][int(np.argmin(inner["rmsecv"]))])        inner_val_original = [outer_train[val] for _, val in inner["folds"]]        pipe = Pipeline(            [                ("scaler", StandardScaler()),                ("pls", PLSRegression(n_components=a_star, scale=False)),            ]        )        pipe.fit(X[outer_train], y[outer_train])        y_oof[outer_val] = np.asarray(            pipe.predict(X[outer_val]), dtype=float        ).reshape(-1)        records.append(            {                "outer_train": outer_train.copy(),                "outer_val": outer_val.copy(),                "inner_val_original": inner_val_original,                "a_star": a_star,            }        )    return {        "y_oof": y_oof,        "rmsecv": pooled_rmsecv(y, y_oof),        "records": records,    }

Practical notes

CV estimates depend on the splitting strategy. The design should represent the intended prediction problem.
Related observations (replicates, batches, subjects, lots) should not be split as if they were independent when the generalization unit is the group.
Transformations that learn parameters from a collection of rows must be fitted on the training fold. A row-wise formula that uses only the current observation does not use other rows.
Choosing components, variables, or preprocessing with CV is model selection. That optimized CV value is not automatically an independent performance estimate.
Nested (double) CV or an untouched external test set is required when the full selection procedure must be assessed.
Pooled RMSECV (this page) is not identical to the mean of fold RMSEs used by sklearn cross_val_score.
cross_val_predict is for out-of-fold visualization, not a drop-in generalization-error estimator.
There is no universal K. Library defaults (5 in sklearn KFold, 10 in MATLAB cvpartition) are not scientific rules.
CV does not correct distribution shift, poor sampling, unrepresentative calibration data, incorrect grouping, leakage, or selection bias by existing.

References

1.
Westad, F., & Marini, F. (2015). Validation of chemometric models. A tutorial. Analytica Chimica Acta, 893, 14-24.
doi:10.1016/j.aca.2015.06.056
2.
Anderssen, E., Dyrstad, K., Westad, F., & Martens, H. (2006). Reducing over-optimism in variable selection by cross-model validation. Chemometrics and Intelligent Laboratory Systems, 84(1-2), 69-74. Abstract and publisher introduction inspected.
doi:10.1016/j.chemolab.2006.04.021
3.
Filzmoser, P., Liebmann, B., & Varmuza, K. (2009). Repeated double cross validation. Journal of Chemometrics, 23(4), 160-171. Abstract inspected.
doi:10.1002/cem.1225
4.
scikit-learn Developers (1.6.1 / 1.9 user guide). Cross-validation, KFold, Pipeline, cross_validate, cross_val_score, and cross_val_predict. scikit-learn documentation.
5.
The MathWorks, Inc. (n.d.). cvpartition, training, and test. MATLAB documentation. Default KFold k is 10. Partition is random unless a custom or consecutive construction is used.

← Back to Open Lab