Mathematical Foundations

II.8 · Linear Algebra

A matrix is a rule for moving every vector—and the whole coordinate grid.

Drag the two image columns. The same exact rule moves an arbitrary vector, reshapes a unit square, preserves or reverses orientation, and sometimes collapses the plane.

Drag the matrix columns; reshape the coordinate paper

This is linear and fixes the origin. Its columns are the two basis images; det A is 1, so orientation is preserved.

matrix A
2×2 [1, 1; 0, 1]
first column / image of e₁
(1, 0)
second column / image of e₂
(1, 1)
input vector
(2, 1)
output vector
(3, 1)
image of the origin
(0, 0)
orientation and area scale
1

Exact expressions use =. Rounded computations use ≈. Every view uses this same state.

This is linear and fixes the origin. Its columns are the two basis images; det A is 1, so orientation is preserved.

A=[Ae1;Ae2]=[1101]A=[A\mathbf e_1;A\mathbf e_2]=\begin{bmatrix}1&1\\0&1\end{bmatrix}
Ax=[31]A\mathbf x=\begin{bmatrix}3\\1\end{bmatrix}
A0=0A\mathbf0=\mathbf0

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

Drag either image column or the input vector. Every line of the coordinate paper follows the same map. Compare a quarter-turn, reflection, unequal scaling, singular collapse, and affine shift.

Begin with the Shear preset. Move x\mathbf x while leaving the columns fixed: the output follows the deformed grid. Then move Ae1A\mathbf e_1 itself. Predict which family of grid lines changes before you drag. Finally choose Affine shift and inspect where the origin goes.

Intuition: specify two directions, determine the whole plane

Every vector in the plane is a weighted combination of the standard basis vectors,

x=x1e1+x2e2.\mathbf x=x_1\mathbf e_1+x_2\mathbf e_2.

A linear transformation respects those same weights. Once we choose the two images Ae1A\mathbf e_1 and Ae2A\mathbf e_2, every other output is forced. The stage draws those images as draggable columns and transforms the complete grid with them.

Definition

A transformation T:R2→R2T:\mathbb R^2\to\mathbb R^2 is linear when

T(u+v)=T(u)+T(v),T(cu)=cT(u).T(\mathbf u+\mathbf v)=T(\mathbf u)+T(\mathbf v), \qquad T(c\mathbf u)=cT(\mathbf u).

Writing the two image vectors as columns gives a matrix

A=[a11a12a21a22]=[Ae1Ae2].A= \begin{bmatrix} a_{11}&a_{12}\\ a_{21}&a_{22} \end{bmatrix} = \begin{bmatrix} A\mathbf e_1&A\mathbf e_2 \end{bmatrix}.

The transformation is then T(x)=AxT(\mathbf x)=A\mathbf x. A linear map always fixes the origin because A0=0A\mathbf0=\mathbf0.

Mechanics: read the transformed grid

Vertical grid lines consist of points with one fixed x1x_1 weight while x2x_2 varies. Their images remain parallel to the second matrix column. Horizontal grid lines remain parallel to the first column. A shear slants one family; unequal scaling changes spacing; a rotation turns both families; a reflection changes handedness.

When the columns become dependent, two grid directions collapse onto one direction. Distinct inputs can then share the same output. The stage keeps this exact singular case visible instead of nudging the columns apart.

Derivation: the columns control every vector

Apply linearity to the basis expansion:

Ax=A(x1e1+x2e2)=x1Ae1+x2Ae2=[a11x1+a12x2a21x1+a22x2].\begin{aligned} A\mathbf x &=A(x_1\mathbf e_1+x_2\mathbf e_2)\\ &=x_1A\mathbf e_1+x_2A\mathbf e_2\\ &= \begin{bmatrix} a_{11}x_1+a_{12}x_2\\ a_{21}x_1+a_{22}x_2 \end{bmatrix}. \end{aligned}

This is why dragging a column moves the grid and the arbitrary output through one state. The picture is not a separate geometric approximation of the formula.

Linear versus affine

An affine transformation has the form

x⟼Ax+b.\mathbf x\longmapsto A\mathbf x+\mathbf b.

It keeps the parallel-line deformation supplied by AA, then translates every output by the same b\mathbf b. If b≠0\mathbf b\ne\mathbf0, the origin moves and the full map is not linear. The stage marks that moved origin explicitly; it does not hide translation inside an extra matrix coordinate.

Worked examples

For

A=[1101],x=[21],A=\begin{bmatrix}1&1\\0&1\end{bmatrix}, \qquad \mathbf x=\begin{bmatrix}2\\1\end{bmatrix},

the columns are (1,0)(1,0) and (1,1)(1,1). Therefore Ax=2(1,0)+(1,1)=(3,1)A\mathbf x=2(1,0)+(1,1)=(3,1). The unit square becomes a slanted parallelogram, but its two directions remain independent.

Inspect the worked calculation

Read the columns as Ae₁ and Ae₂.

A=[Ae1;Ae2]=[1101]A=[A\mathbf e_1;A\mathbf e_2]=\begin{bmatrix}1&1\\0&1\end{bmatrix}
matrix A
2×2 [1, 1; 0, 1]
first column / image of e₁
(1, 0)
second column / image of e₂
(1, 1)
input vector
(2, 1)
output vector
(3, 1)
image of the origin
(0, 0)
orientation and area scale
1
Step from the two basis images to the weighted output. The trace is immutable and uses the same exact model as the live stage.

The quarter-turn preset sends e1\mathbf e_1 upward and e2\mathbf e_2 left. The reflection preset reverses one basis direction. The singular preset uses proportional columns, so the transformed square has zero area and the full grid collapses to one direction.

Executable reference code

The pedagogical Python file represents a small matrix by rows and computes exact rational outputs. The NumPy version uses the matrix multiplication operator and checks both the transformed basis and arbitrary vectors.

code/python/matrix_transform.py
from fractions import Fraction as F


def transform(a, x):
    if not a or len(a[0]) != len(x):
        raise ValueError("matrix columns must match the input length")
    return tuple(sum((entry * value for entry, value in zip(row, x)), F(0)) for row in a)


def affine(a, x, translation):
    linear = transform(a, x)
    if len(linear) != len(translation):
        raise ValueError("translation must match the output length")
    return tuple(value + shift for value, shift in zip(linear, translation))


A = ((F(1), F(1)), (F(0), F(1)))
assert transform(A, (F(0), F(0))) == (0, 0)
assert transform(A, (F(2), F(1))) == (3, 1)
assert affine(A, (F(0), F(0)), (F(2), F(-1))) == (2, -1)
code/python/matrix_transform_numpy.py
import numpy as np

A = np.array([[1.0, 1.0], [0.0, 1.0]])
x = np.array([2.0, 1.0])
b = np.array([2.0, -1.0])

np.testing.assert_allclose(A @ x, [3.0, 1.0])
np.testing.assert_allclose(A @ np.zeros(2), np.zeros(2))
np.testing.assert_allclose(A @ np.zeros(2) + b, b)

Explore

Make the first column twice as long without changing the second. Turn both columns by a quarter-turn. Reverse exactly one column. Make the columns equal, then opposite. For each state, predict the grid, output, origin, and orientation before reading the inspector. Add a translation only after you can explain the linear deformation.

Common mistakes

A matrix is not merely a table of coefficients: here its columns are geometric images. Rows do not show where the basis vectors land. A translation is not linear just because it appears next to a matrix. A singular map is still a valid linear map; it simply loses directions. Finally, a transformed grid need not keep right angles or unit spacing.

Numerical considerations

Entered decimals become exact rational values, so exact collapse means exact dependence rather than a hidden epsilon decision. Display coordinates are floating-point visual proxies derived from that state. In measured data, nearly dependent columns can behave sensitively even when they are exactly independent; that conditioning question is distinct from exact singularity.

Engineering and ML connections

Graphics pipelines use matrices for rotation, reflection, scale, and shear, then add translations for placement. A linear layer in a neural network similarly maps input directions to output directions; its bias makes the layer affine. Thinking in columns reveals what each input feature contributes before later chapters study larger systems and rank.

Exercises

calculation

Build an output from columns

A has columns (2, 1) and (−1, 3). Compute Ax for x = (4, 2).

prediction

Test linearity at the origin

A map sends x to Ax + (2, −1). Is it linear?

Choose one answer

prediction

Read a reflection

A reflection across the y-axis sends e₁ to (−1, 0) and e₂ to (0, 1). What happens to orientation?

Choose one answer

calculation

Detect a collapse

For A = [[1, 2], [0.5, 1]], compute det A.

explanation

Separate deformation from shift

Which statement describes x ↦ Ax + b with b ≠ 0?

Choose one answer

Summary and checkpoints

A linear map is determined by the images of a basis, uses the same weights for every vector, and fixes the origin. Check that you can read rotations, reflections, shears, scalings, and collapses from two columns, and distinguish AxA\mathbf x from Ax+bA\mathbf x+\mathbf b. Next we make the matrix–vector arithmetic explicit.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available