Least Squares

When $Ax = b$ has no solution, we find the best approximate solution by minimizing the error:

$$\min_x \|Ax - b\|^2$$

The Normal Equations

The solution satisfies:

$$A^TA\hat{x} = A^Tb$$

This always has a solution (if $A^TA$ is invertible, it’s unique).

Why It Works

We project $b$ onto $C(A)$. The projection $p = A\hat{x}$ is the closest point in the column space to $b$.

Applications

• Line fitting — find the best line through data points
• Curve fitting — find the best polynomial
• Overdetermined systems — more equations than unknowns

Example

Fitting $y = c + dx$ to points $(1,1), (2,2), (3,2)$:

$$A = \begin{pmatrix}1&1\\1&2\\1&3\end{pmatrix}, \quad b = \begin{pmatrix}1\\2\\2\end{pmatrix}$$

Solve $A^TA\hat{x} = A^Tb$ for best $c, d$.