Mathematical Foundations

II.1 · Linear Algebra

Where do two moves take you?

Place one move after another. The destination reveals component addition, the parallelogram reveals why order does not matter, and the arrow between two tips explains subtraction.

Two moves, one destination

Either head-to-tail order ends at (3, 3). The diagonal from u to v is v − u = (−1, 1); reversing that arrow negates both components.

first direction
(2, 1)
second vector
(1, 2)
combined move
(3, 3)
from the tip of u to the tip of v
(−1, 1)
the reverse move
(1, −1)

Exact expressions use =. Rounded computations use ≈. All arrows use these same quantities.

Either head-to-tail order ends at (3, 3). The diagonal from u to v is v − u = (−1, 1); reversing that arrow negates both components.

u+v=[21]+[12]=[33]v+u=[33]\begin{aligned}\mathbf u+\mathbf v&=\begin{bmatrix}2\\1\end{bmatrix}+\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}3\\3\end{bmatrix}\\\mathbf v+\mathbf u&=\begin{bmatrix}3\\3\end{bmatrix}\end{aligned}
v−u=[12]−[21]=[−11]u−v=[1−1]\begin{aligned}\mathbf v-\mathbf u&=\begin{bmatrix}1\\2\end{bmatrix}-\begin{bmatrix}2\\1\end{bmatrix}=\begin{bmatrix}-1\\1\end{bmatrix}\\\mathbf u-\mathbf v&=\begin{bmatrix}1\\-1\end{bmatrix}\end{aligned}

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

Drag either tip, use its arrow keys, or enter components. The two head-to-tail routes meet at the same destination. The arrow from u's tip to v's tip is v − u.

Before reading, predict the destination after swapping the two moves. Then press Swap the two moves. Next choose Opposite moves: the final arrow disappears because the total displacement is zero. The moves still exist; their effects cancel. Try making only one component cancel and observe the axis on which the sum lies.

Intuition: travel by moves

A vector is a displacement, as I.3 Vectors established. Travel two units right and one up, then one right and two up. Your final displacement is three right and three up. Moving the second arrow’s tail to the first arrow’s tip changes its placement, not its components. This head-to-tail picture turns two instructions into one.

Points and moves play different roles. If you begin at point PP, the intermediate point is P+uP+\mathbf u and the destination is P+u+vP+\mathbf u+\mathbf v. The sum u+v\mathbf u+\mathbf v names the combined displacement, independent of PP. Translating an entire journey changes its points but leaves the combined move unchanged.

Definition

For two vectors in R2\mathbb R^2, addition combines matching components:

u+v=[ux+vxuy+vy].\mathbf u+\mathbf v= \begin{bmatrix}u_x+v_x\\u_y+v_y\end{bmatrix}.

The opposite vector is −v=(−vx,−vy)-\mathbf v=(-v_x,-v_y). Subtraction means addition of the opposite:

u−v=u+(−v).\mathbf u-\mathbf v=\mathbf u+(-\mathbf v).

Vector equality means equality of both components. Addition returns another vector of the same dimension. It does not concatenate components, add lengths, or choose a starting point.

Mechanics: two paths and one diagonal

The blue arrow is u\mathbf u; the green arrow is v\mathbf v. The same labels on the translated arrows identify the unchanged moves. Starting with blue then green and starting with green then blue create the two adjacent sides of a parallelogram. Its diagonal from the origin is u+v\mathbf u+\mathbf v.

The purple arrow has a different job. It starts at the tip of u\mathbf u and ends at the tip of v\mathbf v, so its displacement is destination minus departure, v−u\mathbf v-\mathbf u. This arrow generally does not start at the origin. The inspector displays its components, while the picture shows where it is placed.

Derivation: why the paths agree

Each horizontal move contributes to the horizontal total, and each vertical move contributes to the vertical total. This gives the component formula directly. Commutativity follows from real-number addition:

ux+vx=vx+ux,uy+vy=vy+uy.u_x+v_x=v_x+u_x,\qquad u_y+v_y=v_y+u_y.

Thus u+v=v+u\mathbf u+\mathbf v=\mathbf v+\mathbf u. The intermediate points need not agree; equality concerns the final displacement. Repeating this reasoning for three moves gives associativity, (u+v)+w=u+(v+w)(\mathbf u+\mathbf v)+\mathbf w=\mathbf u+(\mathbf v+\mathbf w). The zero vector is the identity, and v+(−v)=0\mathbf v+(-\mathbf v)=\mathbf0.

For points P,Q,RP,Q,R, the head-to-tail rule also gives

(Q−P)+(R−Q)=R−P.(Q-P)+(R-Q)=R-P.

The intermediate point cancels. If every point is translated by a fixed move t\mathbf t, then (Q+t)−(P+t)=Q−P(Q+\mathbf t)-(P+\mathbf t)=Q-P. This is the algebraic reason that arrow placement does not define the vector.

Worked examples

For u=(2,1)\mathbf u=(2,1) and v=(1,2)\mathbf v=(1,2), the sum is (3,3)(3,3) and the difference from the first tip to the second is (−1,1)(-1,1). Inspect the component calculations, then step backward to revisit the inputs.

Inspect the worked calculation

Move u = (2, 1), then v = (1, 2): add the horizontal and vertical moves separately.

u+v=[21]+[12]=[33]v+u=[33]\begin{aligned}\mathbf u+\mathbf v&=\begin{bmatrix}2\\1\end{bmatrix}+\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}3\\3\end{bmatrix}\\\mathbf v+\mathbf u&=\begin{bmatrix}3\\3\end{bmatrix}\end{aligned}
first direction
(2, 1)
second vector
(1, 2)
combined move
(3, 3)
from the tip of u to the tip of v
(−1, 1)
the reverse move
(1, −1)
These are immutable calculation steps. Previous selects an earlier step; Play advances through the explanation.

A second journey starts at P=(−2,1)P=(-2,1), moves by (3,2)(3,2), and ends at Q=(4,−1)Q=(4,-1) after an unknown move. The intermediate point is (1,3)(1,3), so the missing move is (4,−1)−(1,3)=(3,−4)(4,-1)-(1,3)=(3,-4). Checking P+(3,2)+(3,−4)=QP+(3,2)+(3,-4)=Q tests the answer against the journey.

Executable reference code

The pedagogical version keeps chosen decimal components exact with Python fractions. It spells out the two additions and checks the vector shape. The practical version uses NumPy arrays; its ++ means elementwise addition.

code/python/linear_addition.py
"""Decimal inputs are interpreted exactly; a vector has exactly two components."""
from fractions import Fraction


def vector(values):
    if len(values) != 2:
        raise ValueError('Expected two components')
    return tuple(Fraction(str(x)) for x in values)


def add(u, v):
    u, v = vector(u), vector(v)
    return (u[0] + v[0], u[1] + v[1])


def subtract(v, u):
    v, u = vector(v), vector(u)
    return (v[0] - u[0], v[1] - u[1])


if __name__ == '__main__':
    assert add((2, 1), (1, 2)) == (3, 3)
    assert subtract((1, 2), (2, 1)) == (-1, 1)
code/python/linear_addition_numpy.py
import numpy as np

u = np.array([2.0, 1.0])
v = np.array([1.0, 2.0])
sum_vector = u + v          # elementwise; plain Python lists concatenate
move_from_u_to_v = v - u
assert np.array_equal(sum_vector, [3, 3])
assert np.array_equal(move_from_u_to_v, [-1, 1])

Explore

Return to the opening stage. Make u=(3,1)\mathbf u=(3,1) and v=(−3,2)\mathbf v=(-3,2): predict a vertical sum before checking. Make the vectors equal: the arrow between their tips becomes zero, but their sum doubles the move. Finally choose opposite vectors and explain why the parallelogram collapses while the subtraction arrow remains nonzero.

Common mistakes

The length of a sum is generally not the sum of the lengths. For (1,0)+(0,1)(1,0)+(0,1), the result has length 2\sqrt2, while the separate lengths add to 22. Opposite unit vectors give an even stronger counterexample: their sum has length 00. The directions determine how the moves reinforce or cancel.

Numerical considerations

The stage accepts up to two decimal places and computes component arithmetic as exact rational arithmetic. For instance, its decimal 0.1+0.20.1+0.2 equals 0.30.3 exactly under that input contract. Ordinary binary64 arithmetic can instead store a nearby approximation. Subtracting nearly equal measured components can expose input error; exact arithmetic on the entered decimals cannot recover precision absent from the measurement. NumPy’s direct sum may also overflow for sufficiently large finite inputs. The bounded stage avoids overflow; it does not promise that arbitrary floating-point data does.

Engineering and ML connections

Combining velocities, forces, or displacement updates uses this same component rule when the units and coordinate frame agree. Adding an update Δθ\Delta\boldsymbol\theta to model parameters applies one move in parameter space. Adding a bias to a transformed vector will later form an affine map. That operation adds a displacement; it does not preserve the origin in the way a linear transformation does.

Exercises

calculation

Combine two moves

For u = (3, −2), v = (−1, 4), give u + v.

calculation

Which way does the difference point?

The tips are u = (3, −2), v = (−1, 4). Give the arrow from u to v.

construction

Construct the missing move

You start at P = (−2, 1), move by u = (3, 2), then v, and end at Q = (4, −1). Find v.

prediction

Predict the swap

Swap u and v in the head-to-tail construction. What changes?

Choose one answer

explanation

Read the Python

u and v are length-two Python lists. Which expression adds their components?

Choose one answer

Summary and checkpoints

Two moves combine head-to-tail and component by component. The parallelogram explains commutativity; the arrow between tips explains subtraction. Check that you can name both endpoints of a difference arrow, explain translation invariance, and construct a missing move without confusing a point with a vector. In II.2 Scalar multiplication, a scalar changes the size and sign of one move before it is added to another.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available