Inverse (Linear Algebra)

The matrix inverse has a hidden trap that catches almost everyone the first time: it's the linear-algebra equivalent of division, and just like you can't divide by zero, you can't invert a matrix whose determinant is zero. But there's a second, quieter trap — even when the inverse exists, computing it explicitly is often the wrong tool for the job in real ML code.

💡 In one line: The inverse A⁻¹ is the matrix that undoes A — it exists only when det(A) ≠ 0, and in practice you rarely compute it directly because solving equations is faster and more stable than inverting.

What "Inverse" Means

For a square matrix A, its inverse A⁻¹ is the unique matrix satisfying:

A · A⁻¹ = A⁻¹ · A = I

where I is the identity matrix. Just as multiplying a number by its reciprocal gives 1, multiplying a matrix by its inverse gives the "do-nothing" transformation.

The inverse exists only if:

• A is square (n × n) — non-square matrices don't have a true inverse.
• det(A) ≠ 0 — equivalently, A has full rank and its rows/columns are linearly independent.

A matrix that fails this is called singular, and asking for its inverse is asking to "undo" a transformation that already threw information away — there's no way back.

Computing It

2×2 matrix has a closed-form shortcut:

    | a  b |                    1      |  d  -b |
A = | c  d |    A⁻¹  =  ────────────   |        |
                          ad − bc      | -c   a |

Notice ad − bc is exactly the determinant — this is why det(A) = 0 breaks the formula (division by zero).

For larger matrices, the inverse is computed via Gauss-Jordan elimination or through a decomposition (e.g., LU) — never by hand-expanding cofactors beyond small cases.

Why It Matters: Solving Linear Systems

The inverse's main theoretical use is solving equations of the form Ax = b:

Ax = b   ⟹   x = A⁻¹b

This shows up directly in the normal equation for linear regression:

w = (XᵀX)⁻¹ Xᵀy

where X is the feature matrix and y the targets — solving for the optimal weights w in closed form.

The Practical Catch: Avoid Explicit Inversion

Computing A⁻¹ explicitly and then multiplying is mathematically valid but computationally wasteful and numerically less stable than solving the system directly. Libraries provide solvers built exactly for this:

solve() uses more numerically stable decompositions internally and never forms the full inverse matrix, which matters most when A is large or ill-conditioned (technically invertible, but close to singular).

Non-Square Matrices: The Pseudo-Inverse

Most real data matrices in ML aren't square (more samples than features, or vice versa), so the true inverse doesn't exist. The Moore-Penrose pseudo-inverse (A⁺) generalizes the idea — it finds the best approximate solution to Ax = b even when no exact one exists.

Choosing the Right Approach

Whiteboard

Challenges

• Singular matrices — det(A) = 0 means no inverse exists at all; the equation Ax = b either has no solution or infinitely many.
• Ill-conditioning — a matrix can be technically invertible but numerically fragile, amplifying small input errors into large output errors (measured by the condition number).
• Computing inv() when solve() would do — a common inefficiency: forming the full inverse just to multiply it by a vector, instead of solving the system directly.
• Assuming non-square matrices can be inverted — they can't; only the pseudo-inverse applies, and it returns a best-fit answer, not an exact one.

Practical Notes

• Use np.linalg.solve(A, b) instead of np.linalg.inv(A) @ b whenever the goal is solving a system — faster and more numerically stable.
• Check det(A) isn't near zero before inverting; near-zero determinants signal an ill-conditioned matrix.
• Use np.linalg.pinv() for non-square or rank-deficient matrices — this is what least-squares regression relies on under the hood.
• Regularization (e.g., ridge regression's + λI term) is a common fix for near-singular matrices — it nudges the matrix away from singularity before inverting.

Summary

• A⁻¹ is the matrix satisfying A·A⁻¹ = I — it undoes the transformation A applies.
• It exists only for square matrices with det(A) ≠ 0; anything else is singular and cannot be inverted.
• In ML, the inverse appears in closed-form solutions like the linear regression normal equation.
• Prefer solve() over explicit inv() in practice — same math, better numerical stability.
• For non-square matrices, use the pseudo-inverse instead — it gives the best-fit, not an exact, solution.