Linear Discriminant Analysis

This page uses the classical Gaussian classification formulation of LDA.

Linear discriminant analysis is a supervised classifier. Training observations carry class labels. Each class is modeled as a multivariate normal distribution. Class means may differ. The classes share one covariance matrix. Prior class probabilities may enter the assignment rule. A new observation is assigned to the class with the largest posterior probability under that model, or to an equivalent discriminant score.

scikit-learn derives LDA as the shared-covariance special case of quadratic discriminant analysis. MathWorks documents the same Gaussian mixture construction: linear discriminant analysis keeps one covariance matrix for all classes and lets only the means vary. Biancolillo and Marini describe the chemometric form in the same terms: class centroids, a common within-class variance/covariance matrix, prior probabilities, and assignment by the largest posterior probability.

LDA is historically connected to Fisher's 1936 linear discriminant function. Fisher asked for a linear combination of measurements that maximizes the ratio of the between-species mean difference to the within-species standard deviation. That two-class discrimination criterion is not the same object as the later Gaussian generative classifier taught here. This page does not define LDA only as a method that maximizes class separation.

Under the shared-covariance Gaussian model, class assignment has a closed form. The classifier is inherently multiclass. The decision boundary between any two classes is linear in the predictor space: a line in two dimensions and a hyperplane in $p$ dimensions. scikit-learn notes that the method has no mandatory tuning hyperparameter in its basic form.

Those properties do not make LDA universally preferable to PLS-DA, QDA, or other classifiers. They describe the model. In chemometrics the practical question is often whether the shared covariance can be estimated stably from the available spectra.

Let $X\in {\mathbb{R}}^{n\times p}$ contain $n$ observations in rows and $p$ predictor variables in columns. There are $K$ classes $C_1,\dots ,C_K$. For class $k$ the class-conditional model is

$x\mid C_k\sim \mathcal{N}({\mu }_k,\Sigma )$.

The mean vectors ${\mu }_k$ may differ. The covariance matrices do not. The LDA assumption is ${\Sigma }_1={\Sigma }_2=\cdots ={\Sigma }_K=\Sigma$. The classes do not have identical distributions unless their means also coincide.

This is not an independence assumption. LDA models covariance among predictors. Extreme collinearity can still make the estimated covariance singular or poorly conditioned. Those are different statements.

The Gaussian density is the model. It is not a claim that laboratory data are exactly normal. Biancolillo and Marini note that LDA is sometimes still used when normality is imperfect. That remark is not a robustness theorem, and this page does not treat it as one.

From labeled training data the method estimates each class mean and one shared within-class covariance matrix. A prior ${\pi }_k$ is the probability of class $k$ before the new observation $x$ is seen. Bayes' rule then gives

$P(C_k\mid x)\propto p(x\mid C_k){\pi }_k$, where $p(x\mid C_k)$ is the class-conditional density.

Under shared $\Sigma$, comparing those posteriors is equivalent to comparing linear discriminant scores ${\delta }_k(x)$. The predicted class is the class with the largest score. ${\delta }_k(x)$ is a discriminant score. It is not itself a probability.

Priors need not equal the observed training proportions. That choice depends on the modeling setup. scikit-learn infers class proportions from the training data when priors=None. MATLAB fitcdiscr defaults to empirical priors, the training class frequencies, and also documents uniform or user-specified priors. Priors move the decision. If they are chosen from predictive performance, that choice belongs inside model development and validation. Equal priors are not a universal fix for class imbalance.

The class mean estimator is the sample mean of the observations labeled $k$. MathWorks documents that estimator for unweighted data.

The pooled covariance estimator on this page is the unbiased within-class scatter divided by $n-K$, as documented by MathWorks for linear discriminant analysis. That is the page mathematical convention. scikit-learn stores a prior-weighted combination of per-class covariances, and those per-class matrices use the biased (maximum-likelihood) covariance estimator. The two conventions are not identical. Software can also solve the discriminant problem without forming an explicit matrix inverse.

scikit-learn writes the LDA log-posterior, up to a class-independent constant, as the linear function ${\omega }_k^{\mathsf{T}}x+{\omega }_{k0}$ with ${\omega }_k={\Sigma }^{-1}{\mu }_k$ and ${\omega }_{k0}=-\frac{1}{2}{\mu }_k^{\mathsf{T}}{\Sigma }^{-1}{\mu }_k+\log P(y=k)$. The discriminant ${\delta }_k(x)$ below is that function. Expanding the Gaussian log density shows why the quadratic term in $x$ cancels when $\Sigma$ is shared: the remaining comparison between classes $k$ and $l$ is linear in $x$. The set $\{x:{\delta }_k(x)={\delta }_l(x)\}$ is a hyperplane in $p$ dimensions.

Algorithm 1 is the Gaussian classification workflow. It is not a Fisher projection algorithm and not an eigenvalue decomposition. The inverse in Equation 4 may be replaced by a linear solve. The default scikit-learn svd solver does not compute the covariance matrix, which is a numerical implementation detail, not a change of statistical model.

For two classes the same scores are compared. Assign $x$ to $C_1$ if ${\delta }_1(x)>{\delta }_2(x)$, and to $C_2$ if the inequality is reversed, under equal misclassification costs. The boundary is the set where the two scores are equal.

For $K$ classes compute all $K$ scores and apply Equation 5. That is one joint multiclass model. It is not a collection of independent binary LDA models. scikit-learn describes LDA and QDA as inherently multiclass.

If the shared-covariance assumption is dropped and each class keeps its own ${\Sigma }_k$, the classical method is QDA, not LDA. Quadratic terms in $x$ then remain, and the decision surfaces can be quadratic. This page does not teach QDA beyond that distinction. Dixon and Brereton describe LDA through a pooled variance-covariance matrix and QDA through class-specific matrices.

PCA is unsupervised. It does not use class labels. LDA is supervised. It uses class labels to estimate class means, the shared covariance, and the discriminant scores. Do not describe the Gaussian classifier as PCA with labels. Fisher's projection criterion is a different formulation and is not used as the definition on this page.

PLS-DA, as locked on the Open Lab PLS-DA page, uses PLS with class-coded responses and an explicit classification rule. Classical LDA models class distributions through means, a shared covariance matrix, and priors. Neither method is universally better. Biancolillo and Marini note that inverting the within-class covariance is often impractical for spectroscopic matrices with many correlated variables and limited samples, and that PLS-DA is then a common alternative. Næs and Mevik show that the LDA criterion involves an inverse covariance, and that collinearity can degrade classification ability even though the issue is not simply instability of the criterion itself.

Classical empirical covariance inversion can become singular or poorly conditioned when $p$ is large, when variables are strongly correlated, or when $n$ is limited. That is not the statement that LDA cannot work whenever $p>n$. Modern implementations may use a covariance-free solver, shrinkage, a pseudoinverse, diagonal covariance, or prior dimension reduction. MATLAB documents that DiscrimType 'linear' can apply the minimal Gamma regularization needed to invert a singular empirical covariance, and that pseudolinear uses a pseudoinverse. scikit-learn shrinkage is available with the lsqr and eigen solvers. Shrinkage regularizes covariance estimation. It does not universally improve LDA. Any shrinkage or component count chosen from data belongs inside validation.

Chemometric workflows sometimes compress spectra before classical LDA. PCA-LDA is one example. Kemsley compared PCA and PLS compression before LDA on high-dimensional spectroscopic data. This page does not recommend PCA-LDA as a default, and it is not a PCA-LDA tutorial. If PCA or variable selection is used, and predictive performance is assessed by cross-validation, those steps must be fitted inside each training fold. Fitting PCA or selecting wavelengths on all observations, then cross-validating only LDA, leaks validation information into the features.

LDA does not universally require autoscaling. Scaling depends on the predictor representation. Any data-dependent pretreatment, including SNV, MSC, derivatives, scaling, PCA, or variable selection, must be estimated from training data inside the validation design. Training accuracy is not independent predictive performance. Evaluate held-out predictions with the Classification Metrics and Cross-Validation pages.

The first Python listing is an educational implementation of Equations 2 to 5. It uses the page covariance convention $n-K$ and solves $\Sigma w_k={\mu }_k$ instead of forming ${\Sigma }^{-1}$. It is not a replacement for scikit-learn. Class labels from this implementation can match LinearDiscriminantAnalysis on balanced, nonsingular data with equal priors because a global scale factor in $\Sigma$ then cancels in the argmax. Raw scores need not match. sklearn's stored covariance uses a different estimator.

Production classification uses sklearn LinearDiscriminantAnalysis. The default solver is svd. That solver does not compute the covariance matrix. predict assigns the class with the largest discriminant score. predict_proba estimates class probabilities under the model. It does not guarantee calibrated probabilities. The iris demonstration is an executable example. It is not claimed to be byte-identical to Fisher's 1936 table.

MATLAB training uses fitcdiscr with DiscrimType 'linear'. Prediction uses predict. Do not treat older classify as the primary API. sklearn and MATLAB need not return identical labels unless solver, covariance estimator, priors, and regularization are aligned. The educational MATLAB functions follow the page $n-K$ convention.

import numpy as np  def _validate_xy(X, y):    X = np.asarray(X, dtype=float)    y = np.asarray(y).reshape(-1)    if X.ndim != 2:        raise ValueError("X must be a 2D array of samples by variables.")    if X.shape[0] != y.shape[0]:        raise ValueError("X and y must have the same number of observations.")    if min(X.shape) == 0:        raise ValueError("X must have at least one row and one column.")    if not np.all(np.isfinite(X)):        raise ValueError("X must contain only finite values.")    return X, y  def lda_class_means(X, y):    X, y = _validate_xy(X, y)    classes = np.unique(y)    means = np.vstack([X[y == k].mean(axis=0) for k in classes])    counts = np.array([int(np.sum(y == k)) for k in classes], dtype=int)    if np.any(counts < 1):        raise ValueError("Each class must contain at least one observation.")    return classes, means, counts  def lda_pooled_covariance(X, y):    X, y = _validate_xy(X, y)    classes, means, _counts = lda_class_means(X, y)    n, p = X.shape    k_classes = classes.shape[0]    denom = n - k_classes    if denom < 1:        raise ValueError("Pooled covariance requires n > K.")    scatter = np.zeros((p, p), dtype=float)    for i, k in enumerate(classes):        centered = X[y == k] - means[i]        scatter += centered.T @ centered    return scatter / denom  def lda_discriminant_scores(X, means, Sigma, priors):    X = np.asarray(X, dtype=float)    means = np.asarray(means, dtype=float)    Sigma = np.asarray(Sigma, dtype=float)    priors = np.asarray(priors, dtype=float).reshape(-1)    if X.ndim != 2:        raise ValueError("X must be a 2D array of samples by variables.")    if means.ndim != 2 or Sigma.ndim != 2:        raise ValueError("means must be K by p and Sigma must be p by p.")    if priors.shape[0] != means.shape[0]:        raise ValueError("priors must have one entry per class.")    if not np.all(priors > 0):        raise ValueError("priors must be positive.")    if abs(float(priors.sum()) - 1.0) > 1e-8:        raise ValueError("priors must sum to 1.")    weights = np.linalg.solve(Sigma, means.T).T    return X @ weights.T - 0.5 * np.sum(means * weights, axis=1) + np.log(priors)  def fit_lda_gaussian(X, y, priors=None):    X, y = _validate_xy(X, y)    classes, means, counts = lda_class_means(X, y)    if priors is None:        priors = counts / counts.sum()    else:        priors = np.asarray(priors, dtype=float).reshape(-1)        if priors.shape[0] != classes.shape[0]:            raise ValueError("priors must have one entry per class.")        if not np.all(priors > 0):            raise ValueError("priors must be positive.")        priors = priors / priors.sum()    Sigma = lda_pooled_covariance(X, y)    return {        "classes": classes,        "means": means,        "counts": counts,        "priors": priors,        "covariance": Sigma,    }  def predict_lda_gaussian(model, X):    X = np.asarray(X, dtype=float)    if X.ndim != 2:        raise ValueError("X must be a 2D array of samples by variables.")    if X.shape[1] != model["means"].shape[1]:        raise ValueError("X must have the same number of variables as the fitted model.")    scores = lda_discriminant_scores(        X, model["means"], model["covariance"], model["priors"]    )    y_hat = model["classes"][np.argmax(scores, axis=1)]    return y_hat, scores 

1. 1.

   Fisher, R. A. (1936). The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics, 7(2), 179-188.

   doi:10.1111/j.1469-1809.1936.tb02137.x
2. 2.

   Dixon, S. J., & Brereton, R. G. (2009). Comparison of performance of five common classifiers represented as boundary methods: Euclidean Distance to Centroids, Linear Discriminant Analysis, Quadratic Discriminant Analysis, Learning Vector Quantization and Support Vector Machines, as dependent on data structure. Chemometrics and Intelligent Laboratory Systems, 95(1), 1-17.

   doi:10.1016/j.chemolab.2008.07.010
3. 3.

   Næs, T., & Mevik, B.-H. (2001). Understanding the collinearity problem in regression and discriminant analysis. Journal of Chemometrics, 15, 413-426.

   doi:10.1002/cem.676
4. 4.

   Biancolillo, A., & Marini, F. (2018). Chemometric Methods for Spectroscopy-Based Pharmaceutical Analysis. Frontiers in Chemistry, 6, 576.

   doi:10.3389/fchem.2018.00576
5. 5.

   Kemsley, E. K. (1996). Discriminant analysis of high-dimensional data: A comparison of principal components analysis and partial least squares data reduction methods. Chemometrics and Intelligent Laboratory Systems, 33(1), 47-61.

   doi:10.1016/0169-7439(95)00090-9
6. 6.

   scikit-learn Developers (n.d.). LinearDiscriminantAnalysis and Linear and Quadratic Discriminant Analysis. scikit-learn 1.6 documentation.

7. 7.

   The MathWorks, Inc. (n.d.). fitcdiscr, predict (ClassificationDiscriminant), and Creating Discriminant Analysis Model. MATLAB documentation.