Singular Value Decomposition

Singular Value Decomposition is a factorization of a real matrix $A\in {\mathbb{R}}^{m\times n}$ into three factors $A=U\Sigma V^{\mathsf{T}}$.

The columns of $U$ are left singular vectors. The columns of $V$ are right singular vectors. $\Sigma$ is a rectangular diagonal matrix whose diagonal entries are the singular values of $A$. Those singular values are nonnegative and are conventionally ordered from largest to smallest.

Golub and Reinsch give this factorization for a real rectangular matrix and identify the singular values as the nonnegative square roots of the eigenvalues of $A^{\mathsf{T}}A$. The relevant singular vectors form orthonormal sets. They are not, by themselves, chemical factors or latent physical components.

SVD applies to rectangular matrices as well as to square matrices. It exposes rank through the number of nonzero singular values, and it supplies an ordered sequence of rank one contributions that rebuild $A$.

Truncating that sequence gives a lower rank approximation. Eckart and Young treat this as a least squares problem: among matrices of rank at most $k$, the truncated SVD minimizes the sum of squared entrywise differences from $A$, equivalently the Frobenius norm $\Vert A-B{\Vert }_F$.

SVD is also a numerically standard way to obtain principal components when it is applied to a column-centered data matrix. That use is PCA. SVD itself is the matrix factorization, not the statistical method.

Let $r=\min (m,n)$. A reduced, or economy, SVD can be written

$A=U_r{\Sigma }_r{V}_r^{\mathsf{T}}$

with $U_r\in {\mathbb{R}}^{m\times r}$, ${\Sigma }_r\in {\mathbb{R}}^{r\times r}$, and $V_r\in {\mathbb{R}}^{n\times r}$. In this reduced form

$U_r^{\mathsf{T}}{U}_r=I_r$and$V_r^{\mathsf{T}}{V}_r=I_r$.

Those identities say that the retained singular vectors are orthonormal. They do not imply $U_r{U}_r^{\mathsf{T}}=I_m$ when $U_r$ is rectangular and $r<m$. A full SVD uses square orthogonal factors of orders $m$ and $n$, with $\Sigma$ then the same size as $A$.

The singular values satisfy ${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_r\ge 0$. If ${\mathbf{v}}_i$ is a right singular vector and ${\mathbf{u}}_i$ is a left singular vector for ${\sigma }_i$, then

$A^{\mathsf{T}}A{\mathbf{v}}_i={\sigma }_i^2{\mathbf{v}}_i$and$A{A}^{\mathsf{T}}{\mathbf{u}}_i={\sigma }_i^2{\mathbf{u}}_i$.

The number of nonzero singular values equals the exact rank of $A$. In floating point arithmetic, deciding whether a computed singular value is zero requires a tolerance. No single tolerance is appropriate for every matrix.

Equivalently,

$A={\sum }_{i=1}^r{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^{\mathsf{T}}$.

Each term ${\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^{\mathsf{T}}$ is a rank one matrix. Keeping the first $k$ terms gives the rank $k$ approximation

$A_k=U_k{\Sigma }_k{V}_k^{\mathsf{T}}={\sum }_{i=1}^k{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^{\mathsf{T}}$.

The residual $E_k=A-A_k$ is the part of $A$ not represented by that rank $k$ truncation. It is not automatically noise.

If ${\mathbf{u}}_i$ and ${\mathbf{v}}_i$ are replaced together by $-{\mathbf{u}}_i$ and $-{\mathbf{v}}_i$, the rank one term is unchanged. Different implementations can therefore return opposite singular vector signs while representing the same decomposition.

SVD does not center the matrix. Principal Component Analysis is obtained from SVD when the factorization is applied to a column-centered data matrix $X_c$. In the notation of that page, $P=V$ and $T=U\Sigma =X_{c}V$. For sample covariance PCA with denominator $n-1$,

${\lambda }_i={\sigma }_i^2/(n-1)$.

SVD and PCA are not the same concept. SVD is a matrix decomposition. PCA is a data analysis method. SVD is one way to compute PCA when the input has been centered appropriately.

The identities below use the reduced factors unless a full square orthogonal factor is stated. $r=\min (m,n)$.

An explicit 4 by 3 matrix is decomposed with a reduced SVD. A1 and A2 are the rank 1 and rank 2 reconstructions. The bar plot is the Frobenius residual versus rank.

Python uses the official NumPy routine numpy.linalg.svd. With full_matrices=False, the returned factors use reduced shapes based on $K=\min (M,N)$. $U$ contains left singular vectors. $s$ is a one dimensional array of singular values in descending order. Vt is $V^H$. For a real matrix that is $V^{\mathsf{T}}$. It is not $V$.

Reconstruction follows the NumPy identity (U * s) @ Vt. Rank $k$ truncation keeps the first $k$ singular components of those reduced factors.

MATLAB uses the official MathWorks routine svd. The three-output form satisfies A = U*S*V'. Economy size decomposition is requested with "econ". In that form $S$ is square of order $\min (m,n)$. MathWorks documents that corresponding columns of $U$ and $V$ may change sign without changing U*S*V'.

This page does not use sklearn.decomposition.TruncatedSVD. That estimator does not center its input, so it is not a substitute for covariance PCA.

import numpy as np  def reduced_svd(A):    A = np.asarray(A, dtype=float)     if A.ndim != 2:        raise ValueError("A must be a 2D array.")     if not np.all(np.isfinite(A)):        raise ValueError("A must contain only finite values.")     if min(A.shape) == 0:        raise ValueError("A must have at least one row and one column.")     U, s, Vt = np.linalg.svd(A, full_matrices=False)    return U, s, Vt  def reconstruct_svd(U, s, Vt):    U = np.asarray(U, dtype=float)    s = np.asarray(s, dtype=float)    Vt = np.asarray(Vt, dtype=float)    return (U * s) @ Vt  def rank_k_approx(U, s, Vt, k):    U = np.asarray(U, dtype=float)    s = np.asarray(s, dtype=float)    Vt = np.asarray(Vt, dtype=float)    k = int(k)     if k < 1:        raise ValueError("k must be a positive integer.")     k = min(k, s.size)    return (U[:, :k] * s[:k]) @ Vt[:k, :] 

1. 1.

   Golub, G. H., & Reinsch, C. (1970). Singular value decomposition and least squares solutions. Numerische Mathematik, 14, 403-420.

   doi:10.1007/BF02163027
2. 2.

   Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1(3), 211-218.

   doi:10.1007/BF02288367
3. 3.

   Jolliffe, I. T., & Cadima, J. (2016). Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A, 374, 20150202.

   doi:10.1098/rsta.2015.0202
4. 4.

   NumPy Developers (n.d.). numpy.linalg.svd. NumPy documentation.

5. 5.

   The MathWorks, Inc. (n.d.). svd. MATLAB documentation.