Mathematical Foundations

Foundation demo

One mathematical state, many views.

Move a vector and watch its length, direction, coordinates and formulas change together. Try dragging, keyboard controls and numerical input.

Drag the tip of v\mathbf{v}

Coordinate plane from −7 to 7 on both axes; +x points right and +y points up. The vector v runs from the origin (0, 0) to the point (3, 2). Its horizontal component is 3 and its vertical component is 2.

vector
(3, 2)
x-component
3
y-component
2
magnitude
√133.605551…
direction
0.588003 rad
direction in degrees
33.690068°

= exact · ≈ rounded to 6 decimals

v=[32]\mathbf{v}=\begin{bmatrix}3\\2\end{bmatrix}
∥v∥=x2+y2=32+22=13≈3.605551…\begin{aligned}\norm{\mathbf{v}}&=\sqrt{x^{2}+y^{2}}\\&=\sqrt{3^{2}+2^{2}}\\&=\sqrt{13}\\&\approx 3.605551\ldots\end{aligned}
θ=atan2⁡(y,x)=atan2⁡(2,3)≈0.588003 rad≈33.690068∘\begin{aligned}\theta&=\atantwo(y,x)\\&=\atantwo(2,3)\\&\approx 0.588003\ \mathrm{rad}\\&\approx 33.690068^{\circ}\end{aligned}

from −6 to 6

from −6 to 6

Snap to

Show the work: computing ∥v∥\norm{\mathbf{v}}

v=[32]\mathbf{v}=\begin{bmatrix}3\\2\end{bmatrix}

Start from the two components: how far right, how far up.
Drag the tip of the arrow, focus it and press the arrow keys, or type into the fields. The plane, the values, the formulas and the step-by-step work all read the same state.

Experiments to try

Try these small experiments and notice how the visual, numerical and symbolic representations agree.

CapabilityWhere to look
One state, many viewsMove the tip: the plane, the values, the formulas and the work change together.
Three equivalent inputsDrag the tip; focus it and use the arrow keys (Shift moves five steps); or type into the x and y fields.
Exact versus approximateWatch the magnitude row. Try (3,4)(3,4), (2,2)(2,2) and (2.5,1.5)(2.5,1.5).
Undefined is a stateSet both fields to 00: the direction becomes undefined and the arrow disappears.
A visible snap policySwitch Snap to between 1, ½, 0.1 and Free.
Deterministic traceStep through Show the work; going back selects an earlier frame.
Long equationsOn a narrow screen, scroll the longer formula below without widening the page.
Python examplesCompare the step-by-step calculation with the NumPy version below.

Conventions in force

These conventions help you read vectors and formulas throughout the book.

  • The plane has +x+x to the right and +y+y up. Screen coordinates, where yy points down, are converted in exactly one place.
  • Vectors are columns, v=[xy]\mathbf v=\begin{bmatrix}x\\y\end{bmatrix}. Matrices act from the left, v′=Av\mathbf v'=A\mathbf v, and if AA acts first then B(Av)=(BA)vB(A\mathbf v)=(BA)\mathbf v.
  • Angles are in radians; degrees appear only as a secondary display.
  • == means exactly equal and ≈\approx means a rounded approximation. A rounded number is never shown with an equals sign.

Mathematical typography

Inline mathematics sits in the line: the vector v=(3,2)\mathbf v=(3,2) has components x=3x=3 and y=2y=2, and its length is ∥v∥=x2+y2\lVert\mathbf v\rVert=\sqrt{x^2+y^2}. Display mathematics gets its own line:

∥v∥=x2+y2,θ=atan2⁡(y,x).\lVert\mathbf v\rVert=\sqrt{x^{2}+y^{2}},\qquad \theta=\operatorname{atan2}(y,x).

A very long equation must never widen the page. This one is long on purpose, and on a phone it scrolls inside its own box, which becomes a keyboard-reachable region only when it actually overflows:

L(θ)=−1N∑n=1N∑i=1Vyn,i log⁡exp⁡(zn,i)∑j=1Vexp⁡(zn,j),zn=Whn+b\mathcal{L}(\theta)=-\frac{1}{N}\sum_{n=1}^{N}\sum_{i=1}^{V}y_{n,i}\,\log\frac{\exp\left(z_{n,i}\right)}{\sum_{j=1}^{V}\exp\left(z_{n,j}\right)},\qquad \mathbf{z}_n=W\mathbf{h}_n+\mathbf{b}

Exact and approximate

The inspector never lets a rounded value pose as an exact one.

QuantityExact formApproximationShown as
∥(3,2)∥\lVert(3,2)\rVert13\sqrt{13}3.605551…= √13, then ≈ 3.605551…
∥(3,4)∥\lVert(3,4)\rVert55none needed= 5 only
∥(2.5,1.5)∥\lVert(2.5,1.5)\rVertnot claimed2.915476≈ 2.915476 only
direction of (0,0)(0,0)undefinednoneundefined, with the reason

The trailing ”…” appears only when the exact value is known to be irrational. For (2.5,1.5)(2.5,1.5) the model makes no claim about an exact form, so it shows only what it computed.

Run the calculation in Python

The first example works through the calculation step by step. The second uses NumPy. Compare their results and try changing the input values.

code/python/vector_magnitude.pymechanics first
import math

# A 2D vector is stored as two components. Here: v = (3, 2).
x = 3.0
y = 2.0

# Pythagoras: the hypotenuse of the right triangle with legs |x| and |y|.
magnitude = math.sqrt(x * x + y * y)  # ‖v‖ = √(x² + y²)

# atan2 knows the quadrant of (x, y); atan(y / x) does not.
angle = math.atan2(y, x)  # radians, in (−π, π]

print(magnitude, angle)  # 3.605551275463989 0.5880026035475675
code/python/vector_magnitude_numpy.pythen the library form
import numpy as np

# The practical form: once the mechanism is clear, the library does the bookkeeping.
v = np.array([3.0, 2.0])

magnitude = np.linalg.norm(v)  # ‖v‖₂
angle = np.arctan2(v[1], v[0])  # radians, in (−π, π]

print(magnitude, angle)  # 3.605551275463989 0.5880026035475675

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available