Open Lab/Preprocessing · Method

Detrending

A spectrum wise preprocessing method that removes smooth baseline trends by fitting and subtracting a polynomial from each spectrum.

SpectroscopyNIRBaseline CorrectionPolynomialPythonMATLABTutorial

What is Detrending?

Spectral detrending is a preprocessing transformation that removes a smooth baseline trend from each spectrum independently.

Barnes, Dhanoa and Lister introduced detrending for near-infrared diffuse reflectance spectra in order to account for baseline shift and curvilinearity. Their implementation used a second-degree polynomial regression against the spectral coordinate.

The fitted polynomial contribution is subtracted. What remains is the residual spectral signal after that smooth trend has been removed.

Why use it?

Spectra can contain wavelength-dependent baseline variation that is not the spectral structure of primary analytical interest. In diffuse reflectance NIR, Barnes et al. discussed baseline shift and curvilinearity associated with the physical behavior of powdered or densely packed samples.

Detrending models a smooth trend across the spectral axis and subtracts it. This can make a spectrum less dominated by broad baseline variation before further analysis.

The method does not remove all scatter, and it does not isolate pure chemical information. It should not be assumed to improve every calibration, and it is not required in every application.

How does it work?

The operation is row-wise. Each spectrum is processed from its own intensities and from the spectral coordinate. Other spectra are not used.

Take the spectral coordinate, such as wavelength.
Construct a polynomial basis in that coordinate.
Fit the spectrum as a function of the coordinate by least squares.
Estimate the smooth polynomial trend.
Subtract the fitted trend from the measured spectrum.
Return the residual spectrum.

Classical Barnes detrending uses a second-degree polynomial. The default demonstration below follows that choice.

Barnes et al. presented detrending together with Standard Normal Variate. SNV centers and scales each spectrum. Detrending removes a wavelength-dependent polynomial trend. They may be used as a sequence. Detrending itself is not SNV, and the code on this page does not apply SNV.

Mathematics & algorithm

Let $x_{i} (λ_{j})$ be the intensity of spectrum $i$ at spectral coordinate $λ_{j}$ . For quadratic detrending, a second-degree polynomial is fitted to that spectrum.

Mathematical foundation x_{i} (λ_{j}) = a_{i} λ_{j}^{2} + b_{i} λ_{j} + c_{i} + e_{ij} (1) t^_{i} (λ_{j}) = a^_{i} λ_{j}^{2} + b^_{i} λ_{j} + c^_{i} (2) x_{i, DT} (λ_{j}) = x_{i} (λ_{j}) - t^_{i} (λ_{j}) = e^_{ij} (3) P = 11 ⋮ 1 λ_{1} λ_{2} ⋮ λ_{p} λ_{1}^{2} λ_{2}^{2} ⋮ λ_{p}^{2} (4) β^_{i} = argmin_{β} ∥ x_{i} - P β ∥_{2}^{2} (5) i spectrum / sample index j spectral variable index p number of spectral variables λ_{j} spectral coordinate of variable j x_{i} (λ_{j}) measured intensity a_{i}, b_{i}, c_{i} quadratic, linear and constant trend coefficients t^_{i} fitted polynomial trend e_{ij} residual after the polynomial fit P polynomial design matrix for one spectrum β coefficient vector for the polynomial columns of P Interpretation Equation 3 is the detrended spectrum: the residual after subtracting the fitted trend. Equivalently, t^_{i} = P β^_{i} and x_{i, DT} = x_{i} - t^_{i} . The least squares residual is orthogonal to the columns of P .

When wavelength values are available, those coordinates can be used in $P$ . The educational code below uses a normalized coordinate $t \in [- 1, 1]$ if no axis is supplied. That normalization is a numerical convenience. It is not part of the historical Barnes definition. The polynomial trend depends on the chosen spectral coordinate, so the same degree is not automatically equivalent under a change of axis scaling.

Algorithm 1

Spectral Detrending

InputSpectral matrix X ∈ ℝⁿˣᵖ, spectral coordinate λ, polynomial degree d

OutputDetrended matrix $X_{DT}$

01construct polynomial design matrix $P$ from $λ$
02for $i \leftarrow 1$ to $n$ do
03 $x$ ← spectrum $i$
04estimate $β$ by least squares
05 $x \approx P β$
06 $trend \leftarrow P β$
07 $x_{DT} \leftarrow x - trend$
08end for
09return $X_{DT}$

This pseudocode is a SPARKS representation of Equations 1 to 5. It is not a verbatim extract from Barnes et al. The demonstration uses d = 2, the classical second-degree choice.

Visual example

One spectrum contains peaks plus a quadratic trend. The dashed trace is the fitted second-degree polynomial. Subtracting that fit removes the trend and leaves the residual peak structure.

Controlled quadratic trend. Left: observed spectrum and fitted trend. Right: spectrum after subtracting the fit.

Code

The functions below are SPARKS implementations of Equations 1 to 5. They are not copied from Barnes et al. Both operate on a matrix with rows as samples and columns as spectral variables. Each spectrum is fitted independently. The default degree is 2. If no spectral axis is supplied, a normalized coordinate from $- 1$ to $1$ is used. SNV is not applied.

import numpy as np def detrend_spectra(X, x_axis=None, degree=2):    X = np.asarray(X, dtype=float)     if X.ndim != 2:        raise ValueError(            "X must be a 2D array of samples by spectral variables."        )     n_samples, n_variables = X.shape     if x_axis is None:        x_axis = np.linspace(-1.0, 1.0, n_variables)    else:        x_axis = np.asarray(x_axis, dtype=float)     if x_axis.shape != (n_variables,):        raise ValueError(            "x_axis must contain one value per spectral variable."        )     if degree < 0 or degree >= n_variables:        raise ValueError(            "degree must be nonnegative and smaller than the number of variables."        )     P = np.vander(x_axis, N=degree + 1, increasing=True)    X_dt = np.empty_like(X)     for i, spectrum in enumerate(X):        coefficients, *_ = np.linalg.lstsq(P, spectrum, rcond=None)        trend = P @ coefficients        X_dt[i] = spectrum - trend     return X_dt

Practical notes

Classical Barnes detrending uses a second-degree polynomial fitted independently to each spectrum.
The spectral coordinate is part of the model. When wavelengths are available, they can be used. A normalized axis is only a numerical convenience.
A more flexible polynomial can follow more complex baseline shapes. An unnecessarily flexible trend can also remove meaningful spectral structure.
Detrending and SNV address different forms of variation. They may be combined as a sequence when that choice is appropriate for the dataset. They are not the same transformation.
Preprocessing should be evaluated for the specific dataset and analytical objective. Detrending is not required in every application.

References

1.
Barnes, R. J., Dhanoa, M. S., & Lister, S. J. (1989). Standard Normal Variate Transformation and De-Trending of Near-Infrared Diffuse Reflectance Spectra. Applied Spectroscopy, 43(5), 772-777.
doi:10.1366/0003702894202201
2.
Rinnan, Å., van den Berg, F. W. J., & Engelsen, S. B. (2009). Review of the most common pre-processing techniques for near-infrared spectra. TrAC Trends in Analytical Chemistry, 28(10), 1201-1222.
doi:10.1016/j.trac.2009.07.007

← Back to Open Lab