Classification Metrics

Classification metrics summarize different aspects of agreement between known class labels and predicted class labels, or between labels and continuous decision scores. Sokolova and Lapalme treat them as summaries of a confusion matrix. No single metric universally describes classifier performance.

Different metrics emphasize different consequences of false positives, false negatives, class prevalence, and the decision threshold. Powers notes that commonly used summaries carry prevalence and label-bias effects. This page does not designate a best classification metric.

In chemometrics the same quantities evaluate class predictions from methods such as PLS-DA. Szymańska et al. study number of misclassifications, sensitivity, specificity, and AUROC as diagnostic statistics for PLS-DA. The metrics themselves do not belong to one classifier. They evaluate predictions.

The confusion matrix is the foundation. scikit-learn defines $C_{i,j}$ as the number of observations known to be in group $i$ and predicted to be in group $j$. Rows are the actual class. Columns are the predicted class. Wikipedia and other sources may transpose the axes. This page does not.

For binary labels ordered 0, 1, with class 1 as the positive class:

TP is a true positive: actual positive, predicted positive. TN is a true negative. FP is a false positive. FN is a false negative. The meaning of every later binary metric depends on which class is designated positive. Always state that class. On this page it is label 1, matching the PLS-DA 0/1 coding convention.

Accuracy is $(\mathrm{TP}+\mathrm{TN})/N$, with $N=\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}$. Sokolova and Lapalme define it as that ratio. It is the fraction of observations assigned to the correct class under the evaluated sample distribution. It is not a generic statement that the model is good.

Powers writes Rand accuracy as a prevalence-weighted average of recall and inverse recall. A classifier that always predicts the majority class can therefore obtain high accuracy while missing the minority class. Transparent count example: nine actual class 0 and one actual class 1, all predicted 0, gives accuracy 0.9 and sensitivity 0. Accuracy alone is then close to the majority prevalence.

The binary error rate is $(\mathrm{FP}+\mathrm{FN})/N$, which equals $1$ minus accuracy. It is not the false positive rate and not the false negative rate. For PLS-DA, Szymańska et al. use the number of misclassifications $\mathrm{NMC}=\mathrm{FP}+\mathrm{FN}$ as one diagnostic. NMC is a count, not a universal headline metric.

Sensitivity, also recall or true positive rate, is $\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})$. Among actual positives, it is the fraction predicted positive. Powers and Sokolova give that definition. Szymańska et al. use the same diagnostic form, following Altman and Bland.

Specificity, also true negative rate, is $\mathrm{TN}/(\mathrm{TN}+\mathrm{FP})$. Among actual negatives, it is the fraction predicted negative. It is not precision for the negative class, and it is not negative predictive value. The false positive rate is $\mathrm{FP}/(\mathrm{FP}+\mathrm{TN})=1-\mathrm{specificity}$. Optionally, $\mathrm{FNR}=1-\mathrm{sensitivity}$.

Precision, also positive predictive value, is $\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})$. Among observations predicted positive, it is the fraction that are actually positive. Sokolova and Lapalme and scikit-learn use that ratio.

Sensitivity asks: of the actual positives, how many were detected? Precision asks: of the predicted positives, how many were actually positive? They are not interchangeable.

The binary F1 score is the harmonic mean of precision and recall when both exist:

$F_1=2\mathrm{precision}\mathrm{recall}/(\mathrm{precision}+\mathrm{recall})$, equivalently $F_1=2\mathrm{TP}/(2\mathrm{TP}+\mathrm{FP}+\mathrm{FN})$. scikit-learn documents the second form. TN does not appear. Powers emphasizes that F1 can vary while TN changes freely. Binary F1 therefore does not fully describe the four-cell pattern, and it is not universally superior to accuracy.

scikit-learn defines balanced accuracy as the average of recall obtained on each class. In the binary case that is

$\frac{1}{2}\left(\mathrm{sensitivity}+\mathrm{specificity}\right)$.

Ordinary accuracy weights observations by their frequency in the evaluated set. Balanced accuracy gives equal contribution to class-wise recall. sklearn notes that it avoids inflated estimates on imbalanced datasets, and that for balanced data it equals accuracy. It does not completely solve class imbalance. It does not replace a stated positive class, a confusion matrix, or a valid validation design.

sklearn also notes that this binary quantity equals the area under a ROC curve built from hard binary predictions rather than from continuous scores. That is not AUROC computed from ranking scores.

A continuous decision score, such as a PLS-DA dummy-response prediction, becomes a class label only after a decision rule. Changing the threshold changes sensitivity (TPR) and the false positive rate. Powers defines the ROC as TPR plotted against FPR. Szymańska et al. discuss ROC and AUROC for PLS-DA class scores. This page does not draw that curve. The relation is

$\mathrm{ROC}:\mathrm{TPR}(t)\text{versus}\mathrm{FPR}(t)$ as the threshold $t$ varies.

AUROC is the area under that curve. scikit-learn roc_auc_score computes it from prediction scores. It is not the probability that the classifier is correct. It is not accuracy. The usual threshold-sweep construction uses continuous held-out scores. Passing hard labels discards that sweep. Fitted training scores are not predictive AUROC.

Accuracy, sensitivity, specificity, precision, F1, and balanced accuracy are evaluated after scores have been converted to labels at a specified rule. ROC/AUROC summarizes discrimination across thresholds. The curve is still constructed by varying thresholds, so AUROC is not described here as completely threshold-free.

This page does not prescribe 0.5, the Youden index, maximum F1, or maximum accuracy as a universally correct threshold. Threshold choice can depend on class coding, the scientific objective, and the relative cost of FP and FN. If a threshold is optimized, that choice belongs inside the appropriate validation level. See Cross-Validation.

For $C$ classes the confusion matrix is $C\times C$ under the same true-row, predicted-column convention. The diagonal counts correct assignments. Off-diagonal counts misclassifications. Multiclass performance is not reduced to accuracy alone.

Many binary metrics can be computed one-versus-rest: one class is positive and the remaining classes are negative. That yields class-specific precision, recall, and F1. scikit-learn documents that multiclass support for those scores treats the problem as a collection of binary problems, one per label.

Sokolova and Lapalme distinguish macro-averaging (unweighted mean of per-class scores) from micro-averaging (pool $\mathrm{tp},\mathrm{fp},\mathrm{fn}$ then compute the metric). Macro treats classes equally. Micro favors larger classes. scikit-learn adds weighted averaging: the per-class scores are averaged using class support, the number of true instances of each label. None is universally correct. Report the averaging convention with the number.

Algorithm 1 evaluates predictions. It does not fit a classifier. The inputs should come from a validation design that matches the intended claim: out-of-fold predictions, or an untouched external test set. Do not hide validation inside the metric. For ROC/AUROC, use continuous held-out scores across thresholds rather than a single hard-label operating point.

The Python listing is a small deterministic metric check, not a chemical data set. Educational functions return NaN when a denominator is 0. That is the mathematical policy. scikit-learn precision_score, recall_score, and f1_score expose zero_division. The default warns and returns 0. That fallback is software behavior, not the definition of the ratio. sklearn 1.6 has no specificity_score. Compute specificity from TN and FP.

classification_report prints several per-class summaries. It is not the educational implementation. MATLAB uses the same counts and the same NaN policy. ROC/AUROC is not implemented in MATLAB here because it would require a toolbox-specific curve function. The Python sklearn listing uses held-out scores with roc_auc_score.

import numpy as np  def _as_1d_int(values, name):    arr = np.asarray(values)    if arr.ndim != 1:        raise ValueError(name + " must be a 1D array.")    if arr.shape[0] == 0:        raise ValueError(name + " must contain at least one observation.")    if not np.all(np.isfinite(arr.astype(float))):        raise ValueError(name + " must contain only finite values.")    return arr  def _validate_binary_labels(y_true, y_pred, pos_label):    y_true = _as_1d_int(y_true, "y_true")    y_pred = _as_1d_int(y_pred, "y_pred")    if y_true.shape[0] != y_pred.shape[0]:        raise ValueError("y_true and y_pred must have the same length.")    classes = np.unique(np.concatenate([y_true, y_pred]))    if classes.shape[0] > 2:        raise ValueError("Binary metrics require at most two class labels.")    if pos_label not in classes and classes.shape[0] == 2:        raise ValueError("pos_label must be one of the observed class labels.")    if classes.shape[0] == 1:        neg_label = None if classes[0] == pos_label else classes[0]    else:        neg_label = classes[classes != pos_label][0]    return y_true, y_pred, pos_label, neg_label  def _ratio(numerator, denominator):    if denominator == 0:        return float("nan")    return numerator / denominator  def confusion_counts_binary(y_true, y_pred, pos_label=1):    y_true, y_pred, pos_label, neg_label = _validate_binary_labels(        y_true, y_pred, pos_label    )    tp = int(np.sum((y_true == pos_label) & (y_pred == pos_label)))    if neg_label is None:        tn = 0        fp = 0        fn = int(np.sum((y_true == pos_label) & (y_pred != pos_label)))    else:        tn = int(np.sum((y_true == neg_label) & (y_pred == neg_label)))        fp = int(np.sum((y_true == neg_label) & (y_pred == pos_label)))        fn = int(np.sum((y_true == pos_label) & (y_pred == neg_label)))    return {"tp": tp, "tn": tn, "fp": fp, "fn": fn, "pos_label": pos_label}  def classification_metrics_binary(y_true, y_pred, pos_label=1):    cm = confusion_counts_binary(y_true, y_pred, pos_label=pos_label)    tp, tn, fp, fn = cm["tp"], cm["tn"], cm["fp"], cm["fn"]    n = tp + tn + fp + fn    sensitivity = _ratio(tp, tp + fn)    specificity = _ratio(tn, tn + fp)    precision = _ratio(tp, tp + fp)    if np.isnan(sensitivity) or np.isnan(specificity):        balanced_accuracy = float("nan")    else:        balanced_accuracy = 0.5 * (sensitivity + specificity)    return {        **cm,        "n": n,        "nmc": fp + fn,        "accuracy": (tp + tn) / n,        "error_rate": (fp + fn) / n,        "sensitivity": sensitivity,        "specificity": specificity,        "fpr": _ratio(fp, fp + tn),        "precision": precision,        "f1": _ratio(2 * tp, 2 * tp + fp + fn),        "balanced_accuracy": balanced_accuracy,    } 

1. 1.

   Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427-437.

   doi:10.1016/j.ipm.2009.03.002
2. 2.

   Powers, D. M. W. (2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation. Journal of Machine Learning Technologies, 2(1), 37-63.

3. 3.

   Westerhuis, J. A., Hoefsloot, H. C. J., Smit, S., Vis, D. J., Smilde, A. K., van Velzen, E. J. J., van Duijnhoven, J. P. M., & van Dorsten, F. A. (2008). Assessment of PLSDA cross validation. Metabolomics, 4, 81-89.

   doi:10.1007/s11306-007-0099-6
4. 4.

   Szymańska, E., Saccenti, E., Smilde, A. K., & Westerhuis, J. A. (2012). Double-check: validation of diagnostic statistics for PLS-DA models in metabolomics studies. Metabolomics, 8(Suppl 1), 3-16.

   doi:10.1007/s11306-011-0330-3
5. 5.

   Altman, D. G., & Bland, J. M. (1994). Diagnostic tests. 1: Sensitivity and specificity. BMJ, 308(6943), 1552.

   doi:10.1136/bmj.308.6943.1552
6. 6.

   scikit-learn Developers (n.d.). confusion_matrix, accuracy_score, balanced_accuracy_score, precision_score, recall_score, f1_score, and roc_auc_score. scikit-learn 1.6 documentation.