Projections

The projection of $\mathbf{b}$ onto $\mathbf{a}$:

$$\mathbf{p} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}}\mathbf{a} = \frac{\mathbf{a}\mathbf{a}^T}{\mathbf{a}^T\mathbf{a}}\mathbf{b}$$

Projection Matrix

$$P = \frac{\mathbf{a}\mathbf{a}^T}{\mathbf{a}^T\mathbf{a}}$$

Properties: $P^2 = P$, $P^T = P$ (idempotent and symmetric).

Projection onto a Subspace

For projecting onto $C(A)$:

$$P = A(A^TA)^{-1}A^T$$

Key Property

The error $\mathbf{e} = \mathbf{b} - \mathbf{p}$ is orthogonal to the projection:

$$\mathbf{e} \perp \mathbf{p}$$