Regression Metrics

Regression metrics summarize differences between reference responses and model predictions. Westad and Marini treat RMSE and $R^2$ as figures of merit in chemometric validation. Bellon-Maurel and co-authors review SEP, RMSEP, and bias as the quantities commonly used to describe NIR prediction quality. IUPAC notes that an RMSEP is a summary over a test collection. It characterizes that evaluation, not a sample-specific prediction interval.

A metric number has meaning only with its observations, the provenance of the predictions, the stated convention, the physical unit, and the scientific question. The same RMSE formula applied to calibration fitted values, cross-validation predictions, or an external prediction set answers different questions.

This page uses one signed error for every formula, plot, and listing:

$e_i=y_i-{\hat{y}}_i$.

$y_i$ is the reference (observed) response.${\hat{y}}_i$ is the model prediction. The reference is a laboratory or otherwise measured value. It is not automatically a true concentration. Reference methods carry measurement uncertainty, which Westad and Marini include among the contributions to calibration error.

Residuals are $e_i$ computed from a model fitted using those same observations. Prediction errors are $e_i$ for observations that were not used to fit that particular model. The two are the same algebraic difference. They are not the same validation claim. Do not treat residual, prediction error, and measurement error as interchangeable.

MAE is the mean of $|e_i|$. It stays in the unit of $y$. scikit-learn documents it as the mean absolute error with denominator $n$. MAE is a general regression metric. The chemometric sources used here emphasize RMSE/SEP family statistics more than MAE.

MSE is the mean of $e_i^2$, again with denominator $n$. Squaring gives larger absolute errors more weight than MAE. This MSE is a prediction mean square. It is not an unbiased residual-variance estimator and it does not apply a degrees-of-freedom correction.

RMSE is the square root of that MSE. It is in the unit of $y$. It is not "the average error". It is the square root of the mean squared prediction error under this definition. Because of the square, RMSE responds more strongly than MAE to large absolute errors. That is a property of the loss, not a reason to prefer RMSE or MAE in every application.

Chemometric calibration names the same RMSE family by where the predictions came from. Westad and Marini report RMSEC, RMSECV, and RMSEP as distinct figures of merit. They do not display the aggregation formulas used here.

RMSEC is RMSE on calibration observations using fitted values from the model trained on those observations. It describes in-sample calibration fit. Westad and Marini treat it as a baseline that also reflects sample, instrument, and reference uncertainty. It is not an estimate of future prediction error.

RMSECV is RMSE on out-of-fold predictions. This page uses the pooled convention locked on Cross-Validation: each ${\hat{y}}_i$ comes from a model that excluded observation $i$ in that fold, and the squares are averaged over all $n$ calibration objects. That pooled value is not automatically the mean of fold RMSE values.

RMSEP is RMSE on an external prediction set of $n_p$ observations that were not used to fit or select the model. Bellon-Maurel and co-authors write this magnitude, with denominator equal to the number of prediction objects, for an independent test set. IUPAC treats RMSEP as a model-level summary over those test objects, not as a sample-specific standard error. The number is only as relevant as the set is representative of the intended prediction domain. Do not choose components, variables, or preprocessing by minimizing this RMSEP.

There is no mathematical law that RMSEC is smaller than RMSECV or that RMSECV is smaller than RMSEP. Sampling, grouping, noise, reference uncertainty, and distribution shift can reverse a naive ordering.

Bias on this page is the mean signed error $\bar{e}=\frac{1}{n}\sum e_i$ with $e_i=y_i-{\hat{y}}_i$. A positive bias means the references exceed the predictions on average: the model underpredicts. A negative bias means the predictions exceed the references on average: the model overpredicts.

Bellon-Maurel and co-authors define bias as mean prediction minus mean reference. Their signed bias is the negative of this page's bias. RMSE and MAE are unchanged by that sign choice. Bias is not. Report the convention with the number.

Near-zero bias does not imply small scatter. Bellon-Maurel decompose their SEP into a squared bias term and a bias-corrected scatter they call ${\mathrm{SEP}}_c$. A model can have a small mean offset and still have large prediction errors.

Terminology collision: their equation (1) uses the name SEP for what this page calls RMSEP (root mean square of prediction errors, denominator $n$). Their ${\mathrm{SEP}}_c$ is the bias-corrected scatter, also with denominator $n$ in that equation. They note that replacing $n$ by $n-1$ is another common estimator. This page uses $n$ so that, under the same denominator,${\mathrm{RMSEP}}^2={\bar{e}}^2+{\mathrm{SEP}}_c^2$. If a software SEP uses $n-1$, that identity does not hold. IUPAC's sample-specific standard error of prediction is a different quantity and is not displayed here.

The coefficient of determination used here is the scikit-learn definition:

$R^2=1-\frac{{\sum }_i(y_i-{\hat{y}}_i{)}^2}{{\sum }_i(y_i-\bar{y}{)}^2}$.

It compares squared prediction error with the squared deviation of the references from their mean. It is not accuracy, not a prediction-error unit, and not a percent correct. scikit-learn documents that the value can be negative when predictions are worse than predicting the mean of the evaluated references. Do not force plotted or reported $R^2$ into $[0,1]$ for general prediction evaluation.

This $R^2$ is not universally the square of the Pearson correlation between $y$ and $\hat{y}$. For some intercept-including ordinary least-squares fits they coincide. For general predictive evaluation they need not.

Because the denominator is reference variability, a wide response range can produce a high $R^2$ while absolute error remains scientifically important. Bellon-Maurel and co-authors make the related point that prediction-error summaries depend on the range and distribution of the reference values, which is why RMSE and $R^2$ should be read together. scikit-learn states that $R^2$ may not be meaningfully comparable across datasets. Adjusted $R^2$ is omitted: this page is about predictive summaries, not in-sample linear complexity penalties.

The Python listing implements equations 1-11. rmse is the shared calculation. rmsec, rmsecv, and rmsep only name the prediction source. calibration_cv_prediction_rmse fits a scaler and linear model on the calibration block, computes RMSEC from those fitted values, pooled RMSECV from fold-wise refits on calibration only, and RMSEP on a held-out tail that is not used for model selection.

The sklearn tab matches MAE, MSE, RMSE, and $R^2$ for ordinary non-constant references. MATLAB uses the same error sign and the same denominators.

import numpy as npfrom sklearn.linear_model import LinearRegressionfrom sklearn.model_selection import KFoldfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScaler  def _validate_pair(y_reference, y_prediction):    y = np.asarray(y_reference, dtype=float).reshape(-1)    yhat = np.asarray(y_prediction, dtype=float).reshape(-1)    if y.shape[0] != yhat.shape[0]:        raise ValueError("y_reference and y_prediction must have the same length.")    if y.shape[0] == 0:        raise ValueError("y_reference must contain at least one observation.")    if not np.all(np.isfinite(y)) or not np.all(np.isfinite(yhat)):        raise ValueError("y_reference and y_prediction must contain only finite values.")    return y, yhat  def prediction_errors(y_reference, y_prediction):    y, yhat = _validate_pair(y_reference, y_prediction)    return y - yhat  def mae(y_reference, y_prediction):    errors = prediction_errors(y_reference, y_prediction)    return float(np.mean(np.abs(errors)))  def mse(y_reference, y_prediction):    errors = prediction_errors(y_reference, y_prediction)    return float(np.mean(errors ** 2))  def rmse(y_reference, y_prediction):    return float(np.sqrt(mse(y_reference, y_prediction)))  def bias(y_reference, y_prediction):    errors = prediction_errors(y_reference, y_prediction)    return float(np.mean(errors))  def sep_c(y_reference, y_prediction):    errors = prediction_errors(y_reference, y_prediction)    return float(np.sqrt(np.mean((errors - errors.mean()) ** 2)))  def r2(y_reference, y_prediction):    y, yhat = _validate_pair(y_reference, y_prediction)    if y.shape[0] < 2:        return float("nan")    ss_res = np.sum((y - yhat) ** 2)    ss_tot = np.sum((y - y.mean()) ** 2)    if ss_tot == 0:        return float("nan")    return float(1.0 - ss_res / ss_tot)  def regression_metrics(y_reference, y_prediction):    errors = prediction_errors(y_reference, y_prediction)    return {        "errors": errors,        "n": int(errors.shape[0]),        "bias": bias(y_reference, y_prediction),        "mae": mae(y_reference, y_prediction),        "mse": mse(y_reference, y_prediction),        "rmse": rmse(y_reference, y_prediction),        "sep_c": sep_c(y_reference, y_prediction),        "r2": r2(y_reference, y_prediction),    }  def rmsec(y_calibration, yhat_fitted):    return rmse(y_calibration, yhat_fitted)  def rmsecv(y_calibration, yhat_oof):    return rmse(y_calibration, yhat_oof)  def rmsep(y_prediction_set, yhat_prediction_set):    return rmse(y_prediction_set, yhat_prediction_set)  def calibration_cv_prediction_rmse(X, y, n_test=8, n_splits=5):    X = np.asarray(X, dtype=float)    y = np.asarray(y, dtype=float).reshape(-1)    if X.ndim != 2 or X.shape[0] != y.shape[0]:        raise ValueError("X must be 2D with one row per y observation.")    if n_test < 1 or n_test >= y.shape[0]:        raise ValueError("n_test must be between 1 and n-1.")    X_cal, X_test = X[:-n_test], X[-n_test:]    y_cal, y_test = y[:-n_test], y[-n_test:]    pipe = Pipeline(        [            ("scaler", StandardScaler()),            ("model", LinearRegression()),        ]    )    pipe.fit(X_cal, y_cal)    yhat_cal = np.asarray(pipe.predict(X_cal), dtype=float).reshape(-1)    yhat_test = np.asarray(pipe.predict(X_test), dtype=float).reshape(-1)    splitter = KFold(n_splits=n_splits, shuffle=False)    yhat_oof = np.empty(y_cal.shape[0], dtype=float)    for train, val in splitter.split(X_cal):        fold_pipe = Pipeline(            [                ("scaler", StandardScaler()),                ("model", LinearRegression()),            ]        )        fold_pipe.fit(X_cal[train], y_cal[train])        yhat_oof[val] = np.asarray(            fold_pipe.predict(X_cal[val]), dtype=float        ).reshape(-1)    return {        "rmsec": rmsec(y_cal, yhat_cal),        "rmsecv": rmsecv(y_cal, yhat_oof),        "rmsep": rmsep(y_test, yhat_test),        "yhat_cal": yhat_cal,        "yhat_oof": yhat_oof,        "yhat_test": yhat_test,        "y_cal": y_cal,        "y_test": y_test,    } 

1. 1.

   Westad, F., & Marini, F. (2015). Validation of chemometric models. A tutorial. Analytica Chimica Acta, 893, 14-24.

   doi:10.1016/j.aca.2015.06.056
2. 2.

   Bellon-Maurel, V., Fernandez-Ahumada, E., Palagos, B., Roger, J.-M., & McBratney, A. (2010). Critical review of chemometric indicators commonly used for assessing the quality of the prediction of soil attributes by NIR spectroscopy. Trends in Analytical Chemistry, 29(9), 1073-1081.

   doi:10.1016/j.trac.2010.05.006
3. 3.

   Olivieri, A. C., Faber, N. M., Ferre, J., Boque, R., Kalivas, J. H., & Mark, H. (2006). Uncertainty estimation and figures of merit for multivariate calibration (IUPAC Technical Report). Pure and Applied Chemistry, 78(3), 633-661.

   doi:10.1351/pac200678030633
4. 4.

   scikit-learn Developers (1.6.1 / 1.9 user guide). mean_absolute_error, mean_squared_error, root_mean_squared_error, and r2_score. scikit-learn documentation.