Singular Value Decomposition — SVD (Linear Algebra)

SVD has a hidden generality that makes it the most quietly powerful tool in this whole series: eigenvalues and eigenvectors only exist for square matrices, but almost no real data matrix is square (more samples than features, or the reverse). SVD does something similar to eigen-decomposition, but for any matrix, any shape — which is exactly why it, not eigen-decomposition, is what actually runs under the hood of PCA, recommender systems, and a large chunk of modern dimensionality reduction.

💡 In one line: SVD breaks any matrix — square or not — into a rotation, a scaling, and another rotation, revealing its most important directions in order of importance, which is why it's the tool of choice for compressing and understanding real-world data.

The Decomposition

For any matrix A of shape (m, n), SVD writes it as:

A = U Σ Vᵀ

• U — an (m, m) orthogonal matrix; its columns are the left singular vectors.
• Σ — an (m, n) diagonal matrix of singular values, σ₁ ≥ σ₂ ≥ ... ≥ 0, always non-negative and sorted largest to smallest.
• Vᵀ — an (n, n) orthogonal matrix (transposed); its rows are the right singular vectors.

Geometrically: Vᵀ rotates the input space, Σ stretches along the new axes by the singular values, and U rotates the result into the output space. Every linear transformation, no matter its shape, can be described this way.

How SVD Relates to Everything Before It

SVD isn't a separate idea from determinant, rank, and eigenvalues — it's the decomposition that ties them together, generalized to non-square matrices:

This is why SVD generalizes eigen-decomposition rather than replacing it: for square symmetric matrices, the two coincide almost exactly; for everything else — including every real dataset — SVD is the one that's actually defined.

Low-Rank Approximation

The single most useful property of SVD: keeping only the top k singular values (and their corresponding vectors) gives the best possible rank-k approximation of A — proven optimal by the Eckart-Young theorem. Small singular values contribute little to the matrix and can be dropped with minimal loss.

A ≈ Uₖ Σₖ Vₖᵀ     (using only the top k singular values)

This single idea — "most of a matrix's information lives in its top few singular values" — is the mechanism behind image compression, denoising, and dimensionality reduction all at once.

Where SVD Shows Up in AI

• PCA, done properly — PCA is most commonly computed via SVD of the (centered) data matrix directly, rather than by explicitly forming the covariance matrix and eigen-decomposing it — faster and numerically more stable.
• Recommender systems — classic collaborative filtering factorizes a user-item ratings matrix via SVD (or SVD-like matrix factorization) to uncover latent taste/genre dimensions and predict missing ratings.
• Latent Semantic Analysis (LSA) — applies SVD to a word-document matrix in NLP, uncovering latent topics as an early precursor to modern embeddings.
• Model compression — large weight matrices in neural networks can be approximated with low-rank factors to shrink model size, the same underlying idea used in LoRA-style efficient fine-tuning.
• Noise reduction — since noise tends to spread across many small singular values while real signal concentrates in the largest ones, truncating small singular values is a standard denoising technique.

Choosing How Much to Keep

Whiteboard

Challenges

• Choosing k — too few singular values loses real information; too many keeps noise and defeats the purpose of compression. A common approach is keeping enough to capture a target percentage of total variance (sum of squared singular values).
• Cost on very large matrices — full SVD is computationally expensive; truncated/randomized SVD algorithms compute only the top k components directly, without ever forming the full decomposition.
• Sign ambiguity — like eigenvectors, singular vectors have an arbitrary sign; don't read meaning into whether a component's values are mostly positive or negative.
• Confusing SVD with eigen-decomposition — they coincide only for symmetric matrices; assuming singular values are eigenvalues for a general matrix is a common and incorrect shortcut.

Practical Notes

• Use np.linalg.svd(A, full_matrices=False) for the compact form when A isn't square — avoids computing unnecessary extra dimensions.
• For very large matrices, use randomized/truncated SVD (e.g., scikit-learn's TruncatedSVD) rather than full np.linalg.svd, which computes the entire decomposition even if you only need the top few components.
• Inspect the singular value spectrum before picking k — a sharp drop-off is a strong signal for how much of the matrix is genuinely low-rank.
• Prefer SVD over covariance eigen-decomposition for PCA in code — it's the numerically standard approach and avoids explicitly forming AᵀA, which can amplify numerical errors.

Summary

• SVD factors any matrix, square or not, as A = U Σ Vᵀ — rotate, stretch, rotate.
• Singular values are sorted, non-negative, and directly connect to rank, eigenvalues, and determinant from earlier topics.
• The top-k truncation gives the provably best low-rank approximation — the mechanism behind compression, denoising, and dimensionality reduction.
• In AI, SVD powers PCA, recommender systems, LSA, and low-rank model compression techniques like LoRA.