Mathematical Foundations

II.14 · Linear Algebra

How can we simplify a system without changing its answers?

Edit the augmented system and replay exact row operations. Each frame keeps the equations and their matrix together; a zero coefficient row can be either redundant or contradictory.

Simplify the equations, one legal row operation at a time

Original coefficient matrix
[1, 1; 1, −1]
Pivot count / image dimension
2
Independent null directions
0
Null-space basis
none: only the zero vector
Solution set
one solution
Particular solution (free variables 0)
(2, 1)
Original-system verification
(3, 1)

One exact solution (2, 1). Substitution into the ORIGINAL system verifies Ax = b.

A=[111−1]A=\begin{bmatrix}1&1\\1&-1\end{bmatrix}
rank⁡A+dim⁡ker⁡A=2+0=2\operatorname{rank}A+\dim\ker A=2+0=2
Ax=b,b=[31]A\mathbf x=\mathbf b,\quad\mathbf b=\begin{bmatrix}3\\1\end{bmatrix}
A[21]=[31]A\begin{bmatrix}2\\1\end{bmatrix}=\begin{bmatrix}3\\1\end{bmatrix}

Original mathematical state

The original equations. Coefficients are left of the augmentation boundary.

Rows of the current mathematical state; ◆ marks a pivot
R1113
R21−11
(1)x1+(1)x2=3(1)x1+(−1)x2=1\begin{aligned}(1)x_{1}+(1)x_{2}&=3\\(1)x_{1}+(-1)x_{2}&=1\end{aligned}

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

Edit the exact mathematical input, then replay and inspect each immutable state.

Choose Zero pivot: swap rows, then press Next. Choose Skipped pivot column and locate both free variables. Compare Repeated columns with Contradiction: 0 = 1: the coefficient rank is the same, but the answer set changes.

Intuition: replace difficult equations with equivalent ones

An equation is a constraint on an input. Swapping two constraints changes their order, not their intersection. Multiplying a constraint by a nonzero number rewrites the same statement. Adding a multiple of one equation to another can cancel a variable while retaining all the information needed to recover the original equations.

The augmentation bar is a reminder that the right-hand side belongs to the row. Watch it change during every operation; changing coefficients alone would describe a different system.

Definition

For A∈Rm×nA\in\R^{m\times n} and b∈Rmb\in\R^m, elimination rewrites the augmented matrix [A∣b][A\mid b] using three reversible operations:

  1. Ri↔RjR_i\leftrightarrow R_j.
  2. Ri←cRiR_i\leftarrow cR_i, with c≠0c\ne0.
  3. Ri←Ri+cRjR_i\leftarrow R_i+cR_j, with i≠ji\ne j.

A pivot is the first nonzero coefficient in a nonzero echelon row. Echelon pivots step strictly to the right, all entries below each pivot are zero, and zero coefficient rows occur at the bottom. Reduced row-echelon form additionally has unit pivots and zero entries above them.

Mechanics: follow the current equations

The trace stores exact rational matrices, pivot columns and the operation used. Next, Previous, direct step selection and optional Play select immutable mathematical states. The input fields accept decimals with at most two places in [−4,4][-4,4]; rejected text leaves the previous system intact. Keyboard stepping provides the same edits.

The educational model supports one to four equations and unknowns; the explorer offers 2×2, 2×3, 3×2 and 3×3 systems. It is a bounded exact solver, not an unbounded symbolic engine. Pivot selection takes the first remaining nonzero row deterministically. A zero candidate causes a swap or a skipped column, never division by zero.

Derivation: cancellation and back substitution

Suppose the pivot row begins with px1+qx2=up x_1+q x_2=u and another row begins with rx1+sx2=vr x_1+s x_2=v. After dividing the pivot row by nonzero pp, subtract rr copies of it from the other row:

0x1+(s−rq/p)x2=v−ru/p.0x_1+(s-rq/p)x_2=v-ru/p.

The second equation now determines x2x_2 when its coefficient is nonzero. Substitute that value upward to determine x1x_1. With more variables the same reasoning proceeds from the final pivot row upward. Normalizing pivots during forward elimination is convenient here; it is not required by the definition of echelon form.

A skipped pivot column corresponds to a free variable. Setting free variables to zero produces one particular solution, not the entire answer set. The complete consistent set is x=x0+∑jtjzjx=x_0+\sum_j t_jz_j, where each Azj=0Az_j=0 and each tjt_j is arbitrary.

Worked examples

The opening system is x+y=3x+y=3, x−y=1x-y=1. Replace R2R_2 by R2−R1R_2-R_1 to obtain −2y=−2-2y=-2. Scale by −1/2-1/2, giving y=1y=1, then back substitute to get x=2x=2. The trace above replays these actual equations before optionally clearing above the second pivot. Verification uses the original constraints: 2+1=32+1=3 and 2−1=12-1=1.

For y=1y=1, 2x+y=32x+y=3, the first candidate coefficient is zero. Swapping rows gives a usable pivot and solution (1,1)(1,1). For y+2z=1y+2z=1, 2y+4z=22y+4z=2, the first column is skipped and the second row becomes 0=00=0. Both xx and zz are free: (x,y,z)=(s,1−2t,t)(x,y,z)=(s,1-2t,t). If the second RHS becomes 33, that last row instead asserts 0=10=1 and there is no solution.

Replay the worked derivation

Original mathematical state

The original equations. Coefficients are left of the augmentation boundary.

Rows of the current mathematical state; ◆ marks a pivot
R1113
R21−11
(1)x1+(1)x2=3(1)x1+(−1)x2=1\begin{aligned}(1)x_{1}+(1)x_{2}&=3\\(1)x_{1}+(-1)x_{2}&=1\end{aligned}
Replay the canonical worked example, inspecting each exact mathematical frame.

Executable reference code

The Fraction implementation exposes cancellation, guards zero pivots and retains semantic snapshots. The NumPy reference shows the practical full-rank square solve after the mechanism is understood.

code/python/elimination.py
"""Educational elimination: 0-based Python indices, exact Fraction arithmetic."""
from fractions import Fraction

def elimination(a, b):
    rows = [[Fraction(str(x)) for x in row] + [Fraction(str(rhs))] for row, rhs in zip(a, b)]
    n = len(a[0])
    pivot_row = 0
    trace = [tuple(tuple(row) for row in rows)]
    for col in range(n):
        candidate = next((i for i in range(pivot_row, len(rows)) if rows[i][col]), None)
        if candidate is None:
            continue  # Free column; do not divide by a zero pivot.
        rows[pivot_row], rows[candidate] = rows[candidate], rows[pivot_row]
        pivot = rows[pivot_row][col]
        rows[pivot_row] = [x/pivot for x in rows[pivot_row]]
        for i in range(len(rows)):
            if i != pivot_row:
                factor = rows[i][col]
                rows[i] = [x-factor*y for x, y in zip(rows[i], rows[pivot_row])]
        trace.append(tuple(tuple(row) for row in rows))
        pivot_row += 1
        if pivot_row == len(rows):
            break
    return rows, trace

rref, trace = elimination([[0, 1], [2, 1]], [1, 3])
assert rref == [[1, 0, 1], [0, 1, 1]]
assert trace[0] == ((0, 1, 1), (2, 1, 3))
code/python/elimination_numpy.py
import numpy as np
A = np.array([[0., 1.], [2., 1.]])
b = np.array([1., 3.])
x = np.linalg.solve(A, b)  # Solve directly; no explicit inverse.
assert np.allclose(x, [1., 1.])
assert np.allclose(A @ x, b)
# Rectangular/inconsistent systems need least squares, introduced in II.18.

Explore

Replay the fractional preset and confirm that division produces rational coefficients rather than rounded decimals. Change a redundant RHS to create a contradiction. In the tall example, ask whether all three constraints can be satisfied. In the 3×3 example, follow back substitution from the bottom upward. Use Reset to restore the opening equations.

Common mistakes

A zero row is harmless only when its RHS is also zero. A zero candidate is not proof of singularity until row swapping has been considered. Multiplying a row by zero loses a constraint. Swapping columns changes variable order and is not one of the row operations used here. A free variable need not be zero.

Numerical considerations

Every accepted decimal represents an exact rational; even a tiny nonzero pivot contributes rank. Exact fractions can grow, so manageable dimensions are an intentional bound. Numerical Gaussian elimination generally uses pivoting strategies to control roundoff, rather than the first-nonzero rule used for this exact educational trace. A displayed approximation is never the proof that a row is zero.

Engineering and ML connections

Linear constraints appear in circuit analysis, mechanics and parameter estimation. Elimination explains why factorizations solve many such systems efficiently. Reusing a factorization across right-hand sides avoids repeating work; deriving the inverse in II.16 reuses these same row operations.

Exercises

calculation

Back substitute

Solve x + y = 3 and x − y = 1.

calculation

Perform a row operation

In [1, 2 | 3; 2, 5 | 8], replace R₂ by R₂ − 2R₁. Give the new second row.

prediction

A legitimate multiplier

May you multiply an equation row by zero during equivalent row reduction?

Choose one answer

prediction

Skip a column

The first column is zero and the second has a pivot. What does skipping the first column mean?

Choose one answer

prediction

Read the last row

What does [0, 0 | 1] mean?

Choose one answer

Summary and checkpoints

Follow every operation across the whole augmented row, identify pivot and free columns, distinguish 0=00=0 from 0=10=1, and verify answers in the original system. Echelon form exposes structure; back substitution reads solutions from it.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available