Mathematical Foundations

II.13 · Linear Algebra

Each row is a constraint; a solution satisfies all rows at once.

Move two exact lines through unique, parallel, coincident, vertical, and zero-row cases. The same geometry is the matrix equation Ax = b.

Move two exact constraints between one, none, and many solutions

The two exact constraints meet once at (2, 1). Vertical lines require no special mathematical guess; they are ordinary coefficient rows.

matrix A
2×2 [1, 1; 1, −1]
right-hand side
(3, 1)
exact coefficient determinant
−2
system classification
one exact solution
intersection / solution
(2, 1)

Exact expressions use =. Rounded computations use ≈. Every view uses this same state.

The two exact constraints meet once at (2, 1). Vertical lines require no special mathematical guess; they are ordinary coefficient rows.

[111−1][xy]=[31]\begin{bmatrix}1&1\\1&-1\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}3\\1\end{bmatrix}
1x+1y=31x+−1y=1\begin{aligned}1x+1y&=3\\1x+-1y&=1\end{aligned}
(x,y)=(2,1),A[xy]=b(x,y)=(2, 1),\quad A\begin{bmatrix}x\\y\end{bmatrix}=\mathbf b

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

Edit either coefficient row or right-hand side. The lines and exact classifier agree for unique, parallel, coincident, vertical, and contradictory or redundant zero-row systems.

Begin with the unique intersection. Choose Vertical + horizontal to see that vertical lines need no slope special case. Compare Parallel, none with Coincident, many. Finish with the two zero-row presets and explain why 0=00=0 and 0=10=1 behave differently.

Intuition: simultaneous constraints

One linear equation in two unknowns describes every point on a line. A system asks for points that lie on all its lines simultaneously. The possibilities are geometric:

  • two independent lines meet once;
  • distinct parallel lines never meet;
  • coincident lines share infinitely many points.

Writing the equations as rows of one matrix makes those same constraints available to exact algebra and later algorithms.

Definition

The system

a11x+a12y=b1,a21x+a22y=b2\begin{aligned} a_{11}x+a_{12}y&=b_1,\\ a_{21}x+a_{22}y&=b_2 \end{aligned}

is the matrix equation

Ax=b,A=[a11a12a21a22],x=[xy].A\mathbf x=\mathbf b, \qquad A=\begin{bmatrix}a_{11}&a_{12}\\a_{21}&a_{22}\end{bmatrix}, \quad \mathbf x=\begin{bmatrix}x\\y\end{bmatrix}.

Each row is one constraint. A solution is one vector whose row dot-products equal the corresponding entries of b\mathbf b.

Mechanics: classify before reporting a point

For two variables, the coefficient determinant

Δ=a11a22−a12a21\Delta=a_{11}a_{22}-a_{12}a_{21}

separates the unique case. If Δ≠0\Delta\ne0, the two directions differ and there is one exact intersection. If Δ=0\Delta=0, compare the complete equations, including their right-hand sides: consistent proportional equations describe the same line; inconsistent proportions describe parallel lines.

This classification prevents a solver from dividing by zero and then presenting an invented point.

Derivation: the unique intersection

Multiply the first equation by a22a_{22} and the second by a12a_{12}, then subtract:

Δx=b1a22−a12b2.\Delta x=b_1a_{22}-a_{12}b_2.

Similarly,

Δy=a11b2−b1a21.\Delta y=a_{11}b_2-b_1a_{21}.

When Δ≠0\Delta\ne0,

x=b1a22−a12b2Δ,y=a11b2−b1a21Δ.x=\frac{b_1a_{22}-a_{12}b_2}{\Delta}, \qquad y=\frac{a_{11}b_2-b_1a_{21}}{\Delta}.

Substituting the result into both original rows verifies the solution. This exact two-equation derivation prepares the need for a systematic elimination algorithm in II.14 without implementing that chapter here.

Vertical lines and zero rows

The equation x=2x=2 is the ordinary row 1x+0y=21x+0y=2. It needs no slope and no special mathematical representation. Likewise, 0x+0y=00x+0y=0 is true for every point and adds no restriction. But 0x+0y=10x+0y=1 is false for every point and makes the entire system inconsistent.

None versus infinitely many

When left-hand sides are proportional, inspect the right-hand sides. The equations

x+y=1,2x+2y=3x+y=1,\qquad 2x+2y=3

conflict, so there is no solution. Replacing 33 by 22 makes the second equation redundant, so every point on the shared line is a solution. Both have Δ=0\Delta=0; the right-hand sides distinguish them.

Worked examples

For

x+y=3,x−y=1,x+y=3,\qquad x-y=1,

the determinant is −2-2, and the exact formulas give (x,y)=(2,1)(x,y)=(2,1). Both rows verify: 2+1=32+1=3 and 2−1=12-1=1.

Inspect the worked calculation

Write both constraints as A[x,y]ᵀ = b.

[111−1][xy]=[31]\begin{bmatrix}1&1\\1&-1\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}3\\1\end{bmatrix}
matrix A
2×2 [1, 1; 1, −1]
right-hand side
(3, 1)
exact coefficient determinant
−2
system classification
one exact solution
intersection / solution
(2, 1)
Write the constraints as Ax = b, classify them exactly, and verify the unique point in both rows.

For x=2x=2 and y=−3y=-3, the exact intersection is (2,−3)(2,-3). For coincident equations, the stage reports infinitely many solutions rather than choosing an arbitrary representative.

Executable reference code

The Python reference uses exact fractions and cross-products to classify all three outcomes, including zero rows. The NumPy file demonstrates a well-conditioned unique solve and checks residuals, while explicitly classifying singular examples before calling a numerical solver.

code/python/linear_systems.py
from fractions import Fraction as F


def solve2(a, rhs):
    determinant = a[0][0] * a[1][1] - a[0][1] * a[1][0]
    if determinant:
        return "unique", ((rhs[0] * a[1][1] - a[0][1] * rhs[1]) / determinant,
                          (a[0][0] * rhs[1] - rhs[0] * a[1][0]) / determinant)
    augmented_minors = (a[0][0] * rhs[1] - a[1][0] * rhs[0],
                        a[0][1] * rhs[1] - a[1][1] * rhs[0])
    return ("infinite", None) if augmented_minors == (0, 0) else ("none", None)


assert solve2(((F(1), F(1)), (F(1), F(-1))), (F(3), F(1))) == ("unique", (F(2), F(1)))
assert solve2(((F(1), F(1)), (F(2), F(2))), (F(1), F(3)))[0] == "none"
assert solve2(((F(1), F(1)), (F(2), F(2))), (F(1), F(2)))[0] == "infinite"
code/python/linear_systems_numpy.py
import numpy as np

A = np.array([[1.0, 1.0], [1.0, -1.0]])
b = np.array([3.0, 1.0])
np.testing.assert_allclose(np.linalg.solve(A, b), [2.0, 1.0])

parallel = np.array([[1.0, 1.0], [2.0, 2.0]])
assert np.linalg.matrix_rank(parallel) == 1
assert np.linalg.matrix_rank(np.column_stack((parallel, [1.0, 3.0]))) == 2
assert np.linalg.matrix_rank(np.column_stack((parallel, [1.0, 2.0]))) == 1

Explore

Make one line vertical, then slide it by changing only its right-hand side. Turn the second line until it is parallel to the first. Scale an entire equation, including its right-hand side, and confirm the geometric line does not change. Then scale only the left side and explain why the line moves.

Common mistakes

Zero determinant does not by itself mean “no solution”; it means the unique formula cannot apply. Parallel and coincident equations require the right-hand sides. Dividing by an x-coefficient loses vertical constraints when that coefficient is zero. A point that satisfies one row is not a system solution until every row is checked.

Numerical considerations

Exact decimal inputs let the stage distinguish parallel from nearly parallel without an epsilon. In measured floating-point systems, nearly parallel constraints can make the intersection extremely sensitive. Reliable solvers use factorizations, residual checks, and scale-aware diagnostics. A tiny residual alone does not guarantee an accurate solution when the problem is ill-conditioned.

Engineering and ML connections

Constraint systems appear in calibration, circuit models, geometry, fitting, and equilibrium calculations. Normal equations and least squares later address systems that cannot satisfy every data constraint exactly. Linear layers also evaluate many row constraints at once, but solving for an unknown input is a different task from applying a known matrix.

Exercises

calculation

Find one intersection

Solve x + y = 3 and x − y = 1.

calculation

Include a vertical line

Solve x = 2 and y = −3.

prediction

Parallel constraints

x + y = 1 and 2x + 2y = 3 have how many solutions?

Choose one answer

prediction

Coincident constraints

x + y = 1 and 2x + 2y = 2 have how many solutions?

Choose one answer

prediction

Read a contradictory zero row

A system contains the equation 0x + 0y = 1. What follows?

Choose one answer

Summary and checkpoints

A row is a linear constraint and Ax=bA\mathbf x=\mathbf b asks for a vector satisfying every row. Check that you can distinguish one, none, and infinitely many exact solutions; include vertical lines and zero rows; and verify a candidate in the original equations. II.14 develops this reasoning into general Gaussian elimination.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available