Singular Value Decomposition (SVD)

Every $m \times n$ matrix $A$ can be factored as:

$$A = U\Sigma V^T$$

The Three Matrices

• $U$ — $m \times m$ orthogonal matrix (left singular vectors)
• $\Sigma$ — $m \times n$ diagonal matrix (singular values $\sigma_1 \geq \sigma_2 \geq \cdots \geq 0$)
• $V$ — $n \times n$ orthogonal matrix (right singular vectors)

How to Find It

1. Compute $A^TA$ → eigenvalues $\lambda_i$ → $\sigma_i = \sqrt{\lambda_i}$
2. Eigenvectors of $A^TA$ → columns of $V$
3. $U = AV\Sigma^{-1}$ (or eigenvectors of $AA^T$)

Why It’s Universal

• Works for any matrix (not just square, not just symmetric)
• Singular values are always real and non-negative
• $U$ and $V$ are orthogonal (numerically stable)

The Four Subspaces

$$\begin{aligned} C(A) &= \text{span of first } r \text{ columns of } U \\ N(A^T) &= \text{remaining columns of } U \\ C(A^T) &= \text{span of first } r \text{ columns of } V \\ N(A) &= \text{remaining columns of } V \end{aligned}$$

Applications

• Low-rank approximation — keep the largest singular values
• Pseudoinverse — $A^+ = V\Sigma^+ U^T$
• Image compression — discard small singular values
• Principal Component Analysis — the statistical interpretation