Mathematical Foundations

I.3 · Mathematical Language

What does a vector actually store?

A vector is a displacement, and two numbers describe it. From those numbers we read its length with Pythagoras and its direction with atan2, and we find out why the obvious formula for direction quietly fails.

Drag the tip of v\mathbf{v}

Coordinate plane from −7 to 7 on both axes; +x points right and +y points up. The vector v runs from the origin (0, 0) to the point (3, 2). Its horizontal component is 3 and its vertical component is 2.

vector
(3, 2)
x-component
3
y-component
2
magnitude
√133.605551…
direction
0.588003 rad
direction in degrees
33.690068°

= exact · ≈ rounded to 6 decimals

v=[32]\mathbf{v}=\begin{bmatrix}3\\2\end{bmatrix}
∥v∥=x2+y2=32+22=13≈3.605551…\begin{aligned}\norm{\mathbf{v}}&=\sqrt{x^{2}+y^{2}}\\&=\sqrt{3^{2}+2^{2}}\\&=\sqrt{13}\\&\approx 3.605551\ldots\end{aligned}
θ=atan2⁡(y,x)=atan2⁡(2,3)≈0.588003 rad≈33.690068∘\begin{aligned}\theta&=\atantwo(y,x)\\&=\atantwo(2,3)\\&\approx 0.588003\ \mathrm{rad}\\&\approx 33.690068^{\circ}\end{aligned}

from −6 to 6

from −6 to 6

Snap to

Show the work: computing ∥v∥\norm{\mathbf{v}}

v=[32]\mathbf{v}=\begin{bmatrix}3\\2\end{bmatrix}

Start from the two components: how far right, how far up.

Drag the tip of the arrow, use the arrow keys on it, or type into the fields. The two numbers are the whole vector; everything else on the plane and in the table is computed from them.

Try it before reading on. Pull the tip into each quadrant, then onto an axis, then onto the origin. Two numbers xx and yy are all the page stores; the length, the angle and the triangle are derived from them every time. By the end of the chapter you will know why the length is x2+y2\sqrt{x^2+y^2}, why the direction needs a function called atan2⁡\operatorname{atan2} and not just arctan⁡\arctan, and what a vector is not.

Intuition: a vector is a move

Walk three blocks east and two blocks north. That instruction is a displacement: it says how far to move, not where you are. It means the same thing from your front door and from the library; only the place you end up changes.

A vector in the plane is exactly such a move, written as two numbers, a column:

v=[xy]=[32]\mathbf v=\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}3\\2\end{bmatrix}

The first number is the movement along xx (right is positive) and the second is the movement along yy (up is positive). We draw it as an arrow, but the arrow is a picture of the move; the two numbers are the move.

Points and displacements

A point is a place: P=(1,−1)P=(1,-1) names a location. A vector is a move. They are written with the same two numbers, which is the source of a lot of confusion, so it is worth separating them carefully.

The two are tied by two equations: applying a move to a point gives another point, and two points give the move between them.

Q=P+v,v=Q−P.Q=P+\mathbf v,\qquad \mathbf v=Q-P .

Move the start PP, move the tip QQ

Coordinate plane from −9 to 9 on both axes. Arrow from the point minus 2 comma minus 1 to the point 1 comma 1. Its displacement is 3 right and 2 up, length exactly the square root of 13. A dashed grey copy of the same arrow starts at the origin and ends at (3, 2): the same displacement drawn from the origin.

start point (a place)
(−2, −1)
end point (a place)
(1, 1)
displacement (a vector)
(3, 2)
length of v
√133.605551…
direction of v
0.588003 rad

P and Q are points (places). v, ‖v‖ and θ describe the displacement, and do not change when only P moves.

Q=P+v[11]=[−2−1]+[32]\begin{aligned}Q&=P+\mathbf{v}\\\begin{bmatrix}1\\1\end{bmatrix}&=\begin{bmatrix}-2\\-1\end{bmatrix}+\begin{bmatrix}3\\2\end{bmatrix}\end{aligned}
v=Q−P[32]=[11]−[−2−1]\begin{aligned}\mathbf{v}&=Q-P\\\begin{bmatrix}3\\2\end{bmatrix}&=\begin{bmatrix}1\\1\end{bmatrix}-\begin{bmatrix}-2\\-1\end{bmatrix}\end{aligned}

from −4 to 4, up to 6 decimal places

from −4 to 4, up to 6 decimal places

from −4 to 4, up to 6 decimal places

from −4 to 4, up to 6 decimal places

Snap to

Drag P: the whole arrow slides and the displacement v does not change. Drag Q: P stays and v changes. The dashed grey arrow is v drawn from the origin.

Slide the arrow around and watch the inspector. The points PP and QQ change; v\mathbf v, its length and its direction do not. Now move only the tip: the displacement changes, because it is defined as the difference of the two points. An arrow drawn starting at the origin is the special case P=(0,0)P=(0,0), called standard position; it is the one place where the point QQ and the vector v\mathbf v happen to have the same numbers.

Definition

A vector in the plane is an ordered pair of real numbers, written as a column,

v=[xy],\mathbf v=\begin{bmatrix}x\\y\end{bmatrix},

where xx is the horizontal component and yy is the vertical component. Two vectors are equal exactly when both components are equal. The zero vector 0=[00]\mathbf 0=\begin{bmatrix}0\\0\end{bmatrix} is the move that goes nowhere.

Throughout this book the plane has +x+x to the right and +y+y up, vectors are columns, and angles are in radians measured from the +x+x axis, counter-clockwise turns positive and clockwise turns negative (so (1,−1)(1,-1) has direction −π/4-\pi/4). A vector has two derived quantities: its magnitude ∥v∥\lVert\mathbf v\rVert (how long the move is) and, when it is not the zero vector, its direction θ\theta (which way).

Magnitude: Pythagoras on the components

The move v=(x,y)\mathbf v=(x,y) goes xx across and yy up. Those two legs meet at a right angle, so the straight-line length of the arrow is the hypotenuse:

∥v∥=x2+y2.\lVert\mathbf v\rVert=\sqrt{x^{2}+y^{2}} .

Pythagoras’ theorem says the areas of the squares on the legs add up to the area of the square on the hypotenuse. Here that is literal: drag the vector and compare the three squares.

Squares on the sides: x2+y2=∥v∥2x^{2}+y^{2}=\norm{\mathbf{v}}^{2}

A right triangle with legs |x| = 3 and |y| = 2 and the vector (3, 2) as hypotenuse, with a square drawn on each side. The squares on the legs have areas 9 and 4; the square on the vector has area 13, the sum of the other two.

area of the square on the x-leg
9
area of the square on the y-leg
4
area of the square on the arrow
13
its side: the length
√133.605551…

= exact · ≈ rounded to 6 decimals

x2+y2=9+4=13=∥v∥2\begin{aligned}x^{2}+y^{2}&=9+4\\&=13=\norm{\mathbf{v}}^{2}\end{aligned}

from −4 to 4

from −4 to 4

The square on the x-leg has area x², the square on the y-leg has area y², and the square on the arrow has area ‖v‖². The two small areas always add to the large one.

Notice what the inspector says for (3,2)(3,2): the square on the arrow has area 1313, exactly, so the side is 13\sqrt{13} exactly, and 13≈3.605551\sqrt{13}\approx 3.605551 is only a rounding. For (3,4)(3,4) the area is 2525 and the length is exactly 55. The book writes == for the first kind of statement and ≈\approx for the second, and never mixes them.

Two properties follow at once from the formula. The length is never negative, and it is 00 only for the zero vector, because a sum of squares is 00 only when both squares are. And ∥−v∥=∥v∥\lVert -\mathbf v\rVert=\lVert\mathbf v\rVert, since squaring erases signs: reversing a move does not change how long it is.

Direction: the angle from the x-axis

The direction of a nonzero vector is the signed angle θ\theta from the positive xx axis to the arrow: counter-clockwise is positive, clockwise is negative. In the right triangle, tan⁡θ\tan\theta is rise over run, so for a vector pointing into the right half-plane

θ=arctan⁡yx(x>0).\theta=\arctan\frac{y}{x}\qquad (x>0).

That is a perfectly good formula, for a vector pointing right. The trouble is everything else.

Why arctan⁡(y/x)\arctan(y/x) is not enough

Take v=(−3,2)\mathbf v=(-3,2), which points up and to the left. Then y/x=−2/3y/x=-2/3 and arctan⁡(−2/3)≈−0.588\arctan(-2/3)\approx-0.588 radians, which is a direction pointing down and to the right: exactly backwards. The vector −v=(3,−2)-\mathbf v=(3,-2) has the identical ratio −2/3-2/3, so a formula that sees only the ratio cannot tell v\mathbf v from −v-\mathbf v. And arctan⁡\arctan is only able to answer between −π/2-\pi/2 and π/2\pi/2, the right half-plane, so for a vector in the left half-plane it is wrong by exactly half a turn.

Where does arctan⁡(y/x)\arctan(y/x) go wrong?

Jump to

Coordinate plane from −7 to 7 on both axes. A dashed line through the origin carries the vector v and its opposite −v. Where x is not 0, v and −v share the same ratio y/x. The vector (−3, 2) is in quadrant II. A dashed caution-coloured arrow shows where the naive angle points: the opposite way.

vector
(−3, 2)
the ratio y/x
−0.666667
naive angle, atan(y/x)
−0.588003 rad
true direction, atan2(y, x)
2.553590 rad
what atan was missing
π3.141593…

= exact · ≈ rounded to 6 decimals · ⚠ undefined

x < 0 (quadrant II): the vector points into the LEFT half-plane, but arctan can only answer in the right one, so it is half a turn off. atan2 adds π to fix it.

arctan⁡yx=arctan⁡2−3≈arctan⁡(−0.666667)≈−0.588003 rad\begin{aligned}\arctan\frac{y}{x}&=\arctan\frac{2}{-3}\\&\approx\arctan(-0.666667)\\&\approx -0.588003\ \mathrm{rad}\end{aligned}
θ=atan2⁡(2,−3)=arctan⁡yx+π≈−0.588003+3.141593≈2.55359 rad\begin{aligned}\theta&=\atantwo(2,-3)\\&=\arctan\frac{y}{x}+\pi\\&\approx -0.588003+3.141593\\&\approx 2.55359\ \mathrm{rad}\end{aligned}

from −6 to 6

from −6 to 6

Jump between quadrants and axes. The dashed line through the origin carries v and −v, which share the ratio y/x whenever x is not 0 (on the y-axis no ratio exists, although the direction does); the red arrow is where arctan(y/x) points when it disagrees with the true direction.

The function that fixes this is atan2⁡(y,x)\operatorname{atan2}(y,x). It takes the two components separately, so it can see their signs, and returns the angle in (−π,π](-\pi,\pi]. For x>0x>0 it equals arctan⁡(y/x)\arctan(y/x). For x<0x<0 it equals arctan⁡(y/x)+π\arctan(y/x)+\pi when y≥0y\ge0 and arctan⁡(y/x)−π\arctan(y/x)-\pi when y<0y<0. When x=0x=0 a vertical vector has a perfectly valid direction but no defined ratio y/xy/x; atan2⁡\operatorname{atan2} does not divide, so atan2⁡(y,0)=π/2\operatorname{atan2}(y,0)=\pi/2 for y>0y>0 and −π/2-\pi/2 for y<0y<0.

θ=atan2⁡(y,x),atan2⁡(y,x)−arctan⁡yx∈{0, π, −π}(x≠0).\theta=\operatorname{atan2}(y,x),\qquad \operatorname{atan2}(y,x)-\arctan\frac{y}{x}\in\{0,\ \pi,\ -\pi\}\quad(x\neq0).

This identity needs x≠0x\neq0, because y/xy/x must exist. The correction is exactly 00, π\pi or −π-\pi, decided by the signs of xx and yy alone. Here is the whole picture in one table, every cell computed by the same model the stage above uses.

vectorwherey/xatan(y/x)atan2(y, x)verdict
(3, 2)quadrant I0.66670.58800.5880agree
(−3, 2)quadrant II−0.6667−0.58802.5536differ by +π
(−3, −2)quadrant III0.66670.5880−2.5536differ by −π
(3, −2)quadrant IV−0.6667−0.5880−0.5880agree
(−4, 0)the negative x-axis003.1416differ by +π
(0, 4)the positive y-axisundefinedundefined1.5708atan has no answer
(0, −4)the negative y-axisundefinedundefined−1.5708atan has no answer
(0, 0)the originundefinedundefinedundefinedno direction

And there is one vector for which neither formula has a right answer: the zero vector. It has no direction, because it points nowhere. The stage shows this as an undefined state, not as the number 00.

Worked examples

Four vectors, worked step by step: the length first, then the direction. The last one, (0,−5)(0,-5), is the case where y/xy/x does not exist.

Worked examples: length, then direction

Choose a vector

v=[34]\mathbf{v}=\begin{bmatrix}3\\4\end{bmatrix}

Start from the two components: how far right, how far up.

Choose a vector and step through the work. Going back selects an earlier step; nothing is undone.

Executable reference code

Run the Python examples below to follow the calculations step by step. First the mechanics, written out so every step is visible.

code/python/vector_magnitude.pylength, step by step
import math

# A 2D vector is stored as two components. Here: v = (3, 2).
x = 3.0
y = 2.0

# Pythagoras: the hypotenuse of the right triangle with legs |x| and |y|.
magnitude = math.sqrt(x * x + y * y)  # ‖v‖ = √(x² + y²)

# atan2 knows the quadrant of (x, y); atan(y / x) does not.
angle = math.atan2(y, x)  # radians, in (−π, π]

print(magnitude, angle)  # 3.605551275463989 0.5880026035475675
code/python/vector_direction.pynaive versus atan2, and the zero vector
import math

# v = (-3, 2): left of the y-axis, above the x-axis (quadrant II).
x, y = -3.0, 2.0

theta_naive = math.atan(y / x)  # -0.588...: arctan only returns angles between -pi/2 and pi/2
theta = math.atan2(y, x)  # 2.553...: atan2 looks at the signs of BOTH y and x

# -v = (3, -2) has exactly the same ratio y/x, so atan cannot tell v from -v apart.
theta_naive_of_minus_v = math.atan((-y) / (-x))  # the same -0.588... as for v
theta_of_minus_v = math.atan2(-y, -x)  # -0.588...: atan2 DOES see the difference (it is theta - pi)


def direction(x, y):
    """The angle from the +x axis in (-pi, pi], or None for the zero vector (it has no direction)."""
    if x == 0 and y == 0:
        return None
    return math.atan2(y, x)


zero_direction = direction(0.0, 0.0)  # None: the mathematics has no answer
silent_zero = math.atan2(0.0, 0.0)  # 0.0: the library quietly picks a value. Do not mistake it for a direction.

# A vertical vector has no ratio y/x at all, but it has a perfectly good direction.
try:
    math.atan(2.0 / 0.0)
except ZeroDivisionError:
    vertical_naive = "ZeroDivisionError"
vertical_atan2 = math.atan2(2.0, 0.0)  # pi/2

print(theta_naive, theta, theta_of_minus_v, zero_direction, vertical_naive, vertical_atan2)
code/python/vector_displacement.pypoints and displacements
import math

# A displacement vector says "go 3 right and 2 up". It is not a place.
v = (3.0, 2.0)

# Drawn from the origin, the arrow ends at the point (3, 2) ...
tip_from_origin = (0.0 + v[0], 0.0 + v[1])

# ... drawn from another tail P = (1, -1), the SAME displacement ends at a different point Q = P + v.
P = (1.0, -1.0)
tip_from_other_tail = (P[0] + v[0], P[1] + v[1])  # (4.0, 1.0)

# The displacement is recovered from the two points as Q - P: it does not depend on where the arrow starts.
v_from_other_tail = (tip_from_other_tail[0] - P[0], tip_from_other_tail[1] - P[1])  # (3.0, 2.0)

magnitude = math.hypot(*v_from_other_tail)  # the length is a property of v, not of P

print(tip_from_origin, tip_from_other_tail, v_from_other_tail, magnitude)

Then the practical form. NumPy stores many vectors at once, one per row, and works on all of them in a single call. (Rows are a storage layout for a batch; each vector is still a column in the mathematics.)

code/python/vector_magnitude_numpy.pyone vector, library form
import numpy as np

# The practical form: once the mechanism is clear, the library does the bookkeeping.
v = np.array([3.0, 2.0])

magnitude = np.linalg.norm(v)  # ‖v‖₂
angle = np.arctan2(v[1], v[0])  # radians, in (−π, π]

print(magnitude, angle)  # 3.605551275463989 0.5880026035475675
code/python/vector_batch_numpy.pymany vectors at once
import numpy as np

# In data science code a collection of vectors is usually stored one per ROW. (This book writes a single
# vector as a column; the numbers are the same, only the layout differs.)
V = np.array([[3.0, 2.0], [-3.0, 2.0], [-3.0, -2.0], [3.0, -2.0]])

lengths = np.linalg.norm(V, axis=1)  # one norm per row
angles = np.arctan2(V[:, 1], V[:, 0])  # one quadrant-aware angle per row: y first, then x

print(lengths)  # four equal lengths, each sqrt(13)
print(angles)  # one angle per quadrant: I, II, III, IV

Explore: change the parameters

Use the stages above as a laboratory. These questions have definite answers; check them against the inspector.

  • On the Pythagoras stage, which integer vectors have a whole-number length? Try (3,4)(3,4), (4,3)(4,3), (0,4)(0,4). Which others can you find with components up to 44?
  • On the points stage, move PP to the origin. What do QQ and v\mathbf v now have in common?
  • On the direction stage, press −x axis and then +y axis. Which has a ratio y/xy/x and which does not, and what does atan2⁡\operatorname{atan2} return for each?
  • Walk a vector around the origin counter-clockwise through all four quadrants. Where does arctan⁡(y/x)\arctan(y/x) jump, and where does atan2⁡\operatorname{atan2} jump?

Common mistakes

Numerical considerations

Mathematically ∥v∥=x2+y2\lVert\mathbf v\rVert=\sqrt{x^2+y^2}. In floating point, squaring first can lose the answer completely: (10−200)2=10−400(10^{-200})^2=10^{-400} is smaller than the smallest double, and (10200)2(10^{200})^2 is larger than the largest. The library function hypot avoids both by rescaling before it squares.

vector (x, y)√(x² + y²)hypot(x, y)naive formula
(1.0000e+0, 1.0000e+0)1.4142e+01.4142e+0fine
(1.0000e-100, 1.0000e-100)1.4142e-1001.4142e-100fine
(1.0000e-200, 1.0000e-200)01.4142e-200⚠ wrong
(1.0000e+150, 1.0000e+150)1.4142e+1501.4142e+150fine
(1.0000e+200, 1.0000e+200)∞1.4142e+200⚠ wrong

Two more points. Negative zero exists: −0.0-0.0 equals 0.00.0 in comparisons but atan2(0.0, -0.0) returns π\pi while atan2(0.0, 0.0) returns 00. The stages here normalise −0-0 to 00 so the displayed direction is stable. And the angle from atan2 jumps from π\pi to −π-\pi across the negative xx axis; that is a feature of the range, not a change of direction.

Connections to engineering and machine learning

  • Features as vectors. A data point with two measurements is a vector; its length is its distance from the origin, the Euclidean norm that appears everywhere from nearest-neighbour search to regularisation.
  • Normalisation. Dividing a vector by its length gives a unit vector that keeps the direction and discards the scale; cosine similarity compares directions only. The zero vector has no direction, so normalising it needs a guard (the usual eps in code).
  • Robotics and graphics. Headings, velocities and forces are vectors; atan2 is how a robot turns a displacement to a target into a heading, and why it is the standard function in every such library.
  • Gradients. The gradient of a function is a vector; its length is the steepness and its direction is the way uphill.

Exercises

Answer each question and press Check. A wrong answer tells you which mistake it looks like. After one attempt you may open a worked solution.

calculation

A length from components

The vector v has components x = 5 and y = 12. What is its length ‖v‖?

explanation

Exact or rounded?

The vector v = (1, 2) has ‖v‖² = 5. Which statement is correct?

Choose one answer

calculation

Slide the arrow

The displacement v = (3, 2) starts at the point P = (1, −1). Find the tip Q, then slide the arrow so that it starts at P′ = (−2, 4) and find its new tip Q′, and the length ‖v‖² of the displacement afterward.

construction

Another vector of length √13

The vector (3, 2) has length √13. Enter a DIFFERENT vector with integer components and exactly the same length.

prediction

Predict before you run it

For v = (−1, 1), what does arctan(y/x) return? (The true direction is 135°, up and to the left.)

Choose one answer

calculation

Compute atan2

Give atan2(2, −3) in radians, to three decimals. (v = (−3, 2) is in quadrant II.)

construction

Same ratio, opposite way

Build a vector w that has the SAME ratio y/x as v = (−3, 2) but points the opposite way. Enter integers or decimals.

prediction

A tiny vector

In Python, v = (1e-200, 1e-200). What does math.sqrt(x*x + y*y) return, versus math.hypot(x, y)?

Choose one answer

explanation

The zero vector

What does math.atan2(0.0, 0.0) return in Python, and what does that tell you?

Choose one answer

Summary and checkpoints

  • A vector in the plane is a column of two numbers, a displacement: how far, not where. Points are places; Q=P+vQ=P+\mathbf v and v=Q−P\mathbf v=Q-P.
  • Its magnitude is ∥v∥=x2+y2\lVert\mathbf v\rVert=\sqrt{x^2+y^2}, Pythagoras on the components. It is exact when the sum is a perfect square or a known radical, and otherwise a rounding.
  • Its direction is θ=atan2⁡(y,x)\theta=\operatorname{atan2}(y,x), not arctan⁡(y/x)\arctan(y/x), which is wrong by ±π\pm\pi in the left half-plane and does not exist on the yy-axis.
  • The zero vector has length 00 and no direction.
  • In code: hypot for length, atan2 for direction, an explicit guard for zero.

Checkpoints. You are ready to move on when you can, without looking:

  1. Say why (−3,2)(-3,2) and (3,−2)(3,-2) have the same y/xy/x but different directions.
  2. Compute ∥(5,12)∥\lVert(5,12)\rVert and state whether the answer is exact.
  3. Explain why sliding an arrow does not change the vector.
  4. Say what atan2⁡\operatorname{atan2} returns for (0,4)(0,4), (−4,0)(-4,0) and (0,0)(0,0).

You can revisit the interactive vector by itself in the foundation explorer.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available