Mathematical Foundations

IV.1 · Probability

What could happen before we see what did happen?

Select possible outcomes and build an event. The model assigns probability to all possibilities; one observed roll supplies only one result.

An experiment and its sample space

1Original P = 1/6
2Original P = 1/6
3Original P = 1/6
4Original P = 1/6
5Original P = 1/6
6Original P = 1/6

Observed sequence · first 16 at most

  1. 2
  2. 1
  3. 4
  4. 2
  5. 3
  6. 1
  7. 3
  8. 1
  9. 6
  10. 6
  11. 6
  12. 3

Sample space: 1, 2, 3, 4, 5, 6. Equal weights explicitly assume equiprobable outcomes. A = {2, 4, 6}; B = {4, 5, 6}. The fixed model assigns probabilities before a trial. Seed 42 selects this reproducible observed sequence; its frequencies do not redefine the model.

Total probability
1
Event A
0.5
Event B
0.5
Intersection
1/30.333333…
Observed frequency of A
0.5

= exact model quantity · ≈ numerical companion or observed estimate

from 0 to 100, whole numbers only

from 0 to 4294967295, whole numbers only

from 1 to 2000, whole numbers only

The fixed model assigns probabilities before a trial. Seed 42 selects this reproducible observed sequence; its frequencies do not redefine the model.
P(A)=∑ω∈Apω=0.5P(A)=\sum_{\omega\in A}p_\omega=0.5
P(A∪B)=P(A)+P(B)−P(A∩B)=23P(A\cup B)=P(A)+P(B)-P(A\cap B)=\frac{2}{3}

Define the experiment

6 mutually exclusive outcomes, total mass one. Equal weights explicitly assume equiprobable outcomes.

∑ωpω=1\sum_\omega p_\omega=1
1= 1/6
2= 1/6
3= 1/6
4= 1/6
5= 1/6
6= 1/6

Select die outcomes to change event A. Compare its selected faces, individual weights, and total probability; reset to the fair six-sided die to replay the same model.

Before changing anything, predict the mass of the even faces. Select just one face, then all six. Explain why an event can contain several outcomes although a single roll produces only one outcome.

Intuition: possibilities are not observations

A die in your hand has not yet produced a result. A probability model describes what may happen and assigns weights to those possibilities. After a roll, you observe one face. The five other faces remain possibilities for a new roll under the same model; the observation does not erase them from the sample space.

An event answers a question such as “was the face even?” It collects every outcome that would make the answer yes. A probability is the total mass of that collection, not a promise about the next observed result.

Definition

A finite sample space Ω\Omega is a nonempty set of distinct outcomes. A probability model assigns a mass pωp_\omega to each outcome, subject to

pω≥0,∑ω∈Ωpω=1.p_\omega\ge0,\qquad \sum_{\omega\in\Omega}p_\omega=1.

An event AA is a subset of Ω\Omega, with probability P(A)=∑ω∈ApωP(A)=\sum_{\omega\in A}p_\omega. The empty event has probability zero and the whole sample space has probability one. A possible label may have mass zero in a specified model; nonempty does not always mean positive probability.

For the canonical fair six-sided die, Ω={1,2,3,4,5,6}\Omega=\{1,2,3,4,5,6\} and every mass is 1/61/6. Fairness is an explicit assumption, not a consequence of having six labels.

Mechanics: one state, several views

The stage holds the experiment’s outcome weights and event membership. Selecting a face for A changes membership. Edit weight selects a face whose nonnegative raw weight you can change; the model divides all raw weights by their positive total to obtain normalized probabilities. Equal raw weights express the fairness assumption. The selected outcomes, mass inspector, equation, accessible description, and trace refer to that same state.

Read the trace from defining the weighted experiment through its first seeded observation and the bounded repeated-trial counts. Compare those observations with the fixed model mass in the inspector. Replay the same seed to recover the same sequence; changing event membership changes which observations count toward its frequency, not the sequence itself.

Derivation: when counting is enough

If there are nn equally likely outcomes, normalization gives np=1np=1, hence p=1/np=1/n. If AA contains mm of them, then

P(A)=m1n=∣A∣∣Ω∣.P(A)=m\frac1n=\frac{|A|}{|\Omega|}.

This shortcut follows from equal masses. With unequal masses, the set still tells you which terms to add, but its size no longer tells you their sum. Nor does a short observed sequence determine the weights of a future experiment.

Worked examples

For A={2,4,6}A=\{2,4,6\} on the fair die, three selected outcomes each contribute 1/61/6, so P(A)=1/2P(A)=1/2. For B={4,5,6}B=\{4,5,6\}, P(B)=1/2P(B)=1/2 as well. Their shared faces are 44 and 66, with total mass 1/31/3; equal event masses do not make the events identical.

Worked example: An experiment and its sample space

1Original P = 1/6
2Original P = 1/6
3Original P = 1/6
4Original P = 1/6
5Original P = 1/6
6Original P = 1/6

Observed sequence · first 16 at most

  1. 2
  2. 1
  3. 4
  4. 2
  5. 3
  6. 1
  7. 3
  8. 1
  9. 6
  10. 6
  11. 6
  12. 3

Sample space: 1, 2, 3, 4, 5, 6. Equal weights explicitly assume equiprobable outcomes. A = {2, 4, 6}; B = {4, 5, 6}. The fixed model assigns probabilities before a trial. Seed 42 selects this reproducible observed sequence; its frequencies do not redefine the model.

Total probability
1
Event A
0.5
Event B
0.5
Intersection
1/30.333333…
Observed frequency of A
0.5

= exact model quantity · ≈ numerical companion or observed estimate

from 0 to 100, whole numbers only

from 0 to 4294967295, whole numbers only

from 1 to 2000, whole numbers only

The fixed model assigns probabilities before a trial. Seed 42 selects this reproducible observed sequence; its frequencies do not redefine the model.
P(A)=∑ω∈Apω=0.5P(A)=\sum_{\omega\in A}p_\omega=0.5
P(A∪B)=P(A)+P(B)−P(A∩B)=23P(A\cup B)=P(A)+P(B)-P(A\cap B)=\frac{2}{3}

Define the experiment

6 mutually exclusive outcomes, total mass one. Equal weights explicitly assume equiprobable outcomes.

∑ωpω=1\sum_\omega p_\omega=1
1= 1/6
2= 1/6
3= 1/6
4= 1/6
5= 1/6
6= 1/6

Toggle membership in the worked die model and follow the sum of selected 1/6 masses. Restore A = {2, 4, 6} and B = {4, 5, 6} to compare the two half-mass events.

For two independent fair coins, distinguish the first and second positions: Ω={HH,HT,TH,TT}\Omega=\{HH,HT,TH,TT\}, each mass 1/41/4. The event “exactly one head” contains HTHT and THTH, so its mass is 1/21/2. If we describe only the number of heads, the labels 0,1,20,1,2 have masses 1/4,1/2,1/41/4,1/2,1/4. Grouping outcomes changes the labels, not the total mass associated with each group.

Executable reference code

The standard-library program spells out exact Fraction masses, validates the finite model, sums event weights, and prints the die and coin examples. Its bounded observation sequence uses an explicit seed. The NumPy program compares boolean-mask sums against independently specified exact fractions; it performs no random experiment to manufacture an answer.

code/python/probability_experiments.py
"""A finite probability model is not an observed sequence of trials.

Fractions made from integers keep these model probabilities exact. The seeded,
bounded die sequence illustrates observations; its frequency is not the model.
"""
from collections import Counter
from fractions import Fraction as F
from random import Random


def validate_model(weights):
    if not weights or any(not isinstance(p, F) or p < 0 for p in weights.values()):
        raise ValueError("Use a nonempty outcome map with nonnegative Fraction masses")
    if sum(weights.values(), F(0)) != 1:
        raise ValueError("Outcome masses must sum exactly to one")


def event_mass(weights, event):
    validate_model(weights)
    if not set(event) <= weights.keys():
        raise ValueError("An event must be a subset of the sample space")
    return sum((weights[outcome] for outcome in set(event)), F(0))


def main():
    die = {face: F(1, 6) for face in range(1, 7)}
    even = {2, 4, 6}
    assert event_mass(die, even) == F(1, 2)
    assert event_mass(die, set()) == 0
    assert event_mass(die, set(die)) == 1
    coins = {outcome: F(1, 4) for outcome in ("HH", "HT", "TH", "TT")}
    assert event_mass(coins, {"HT", "TH"}) == F(1, 2)
    weighted = {"red": F(1, 2), "green": F(1, 3), "blue": F(1, 6)}
    assert event_mass(weighted, {"green", "blue"}) == F(1, 2)
    for bad in ({}, {1: F(1, 2)}, {1: F(-1, 4), 2: F(5, 4)}):
        try:
            validate_model(bad)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid model was accepted")
    try:
        event_mass(die, {7})
    except ValueError:
        pass
    else:
        raise AssertionError("An outcome outside the sample space was accepted")

    # The random generator samples the fair-die model, not arbitrary weights.
    def observations(seed, count):
        rng = Random(seed)
        return tuple(rng.randrange(1, 7) for _ in range(count))

    rolls = observations(2026, 24)
    assert rolls == observations(2026, 24)
    counts = Counter(rolls)
    observed_even = F(sum(counts[face] for face in even), len(rolls))
    print("Fair die: Ω =", tuple(die), "; exact P(even) =", event_mass(die, even))
    print("Two fair independent coins: exact P(one head) =", event_mass(coins, {"HT", "TH"}))
    print("Unequal weights: exact P(green or blue) =", event_mass(weighted, {"green", "blue"}))
    print("24 observed rolls, seed 2026:", rolls)
    print("Empirical even frequency =", observed_even, "(not a replacement for P(even) = 1/2)")


if __name__ == "__main__":
    main()
code/python/probability_experiments_numpy.py
"""Finite experiment masses as arrays; NumPy decimals approximate exact fractions.

Small-array comparisons use absolute tolerance 1e-12 and zero relative tolerance.
The tolerance checks arithmetic, not whether a physical experiment is fair.
"""
from fractions import Fraction as F
import numpy as np

ATOL = 1e-12


def validate_weights(weights):
    p = np.asarray(weights, dtype=float)
    if p.ndim != 1 or p.size == 0 or not np.all(np.isfinite(p)) or np.any(p < 0):
        raise ValueError("Use a nonempty finite nonnegative mass vector")
    if not np.isclose(p.sum(), 1.0, rtol=0, atol=ATOL):
        raise ValueError("Mass vector must sum to one within floating-point tolerance")
    return p  # No clipping or silent normalization.


def main():
    faces = np.arange(1, 7)
    p = validate_weights(np.full(6, 1 / 6))
    even = faces % 2 == 0
    even_mass = p[even].sum()
    np.testing.assert_allclose(even_mass, float(F(1, 2)), rtol=0, atol=ATOL)
    np.testing.assert_allclose(p[faces == 6].sum(), float(F(1, 6)), rtol=0, atol=ATOL)
    assert p[np.zeros(6, dtype=bool)].sum() == 0
    coins = np.array(["HH", "HT", "TH", "TT"])
    coin_mass = validate_weights(np.full(4, 1 / 4))
    one_head = np.isin(coins, ["HT", "TH"])
    np.testing.assert_allclose(coin_mass[one_head].sum(), float(F(1, 2)), rtol=0, atol=ATOL)
    # Counting heads merges HT and TH into a group with twice the other masses.
    head_counts = np.array([2, 1, 1, 0])
    grouped = np.bincount(head_counts, weights=coin_mass, minlength=3)
    np.testing.assert_allclose(grouped, [float(F(1, 4)), float(F(1, 2)), float(F(1, 4))], rtol=0, atol=ATOL)
    weighted = validate_weights([float(F(1, 2)), float(F(1, 3)), float(F(1, 6))])
    np.testing.assert_allclose(weighted[[1, 2]].sum(), float(F(1, 2)), rtol=0, atol=ATOL)
    for bad in ([], [0.5, 0.4], [-0.25, 1.25], [np.nan, 1.0], [np.inf], [[1.0]]):
        try:
            validate_weights(bad)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid mass vector was accepted")
    print("Die event mask:", even, "; P(even) ≈", even_mass, "(exact 1/2)")
    print("Two-coin head-count masses ≈", grouped, "(exact 1/4, 1/2, 1/4)")
    print("Weighted green-or-blue mass ≈", weighted[[1, 2]].sum(), "(exact 1/2)")


if __name__ == "__main__":
    main()

Explore

Construct the singleton event containing 66, the empty event, and the whole die space. Predict 1/61/6, 00, and 11 before inspecting the answers. Return to the even faces and compare their model mass with the observed frequency. Change Experiment seed, increase Observed trials, then use Replay seed. Neither setting changes the model probabilities. The stage’s display shows at most the first 16 outcomes while its frequency includes all requested trials, bounded at 2000.

Choose Weighted die or edit a raw weight to break equiprobability. In the weighted reference example, compare two selected labels with half the total mass: the number of labels is not the denominator of a universal probability rule. Choose Two fair coins to inspect the four ordered outcomes.

Common mistakes and counterexamples

Three even observations in a row do not imply that the model now contains only even faces. A fair independent die does not owe you an odd face after a streak. Conversely, a streak does not prove a die fair: fairness is stipulated here; estimating a physical die’s probabilities from data is a later statistical problem.

For a counterexample to counting alone, assign red, green, and blue masses 1/2,1/3,1/61/2,1/3,1/6. Selecting green and blue gives 1/21/2, although two of three labels are selected. Treating those labels as equally likely silently replaces the model.

Numerical considerations

The finite examples have exact rational answers. Python constructs Fraction(1, 6) from integers; constructing a fraction from a binary floating-point approximation would preserve that approximation instead. NumPy arrays use floating-point masses and sums, so the reference marks these comparisons as approximate and checks absolute error with an explicit 10−1210^{-12} tolerance, not a claim of exact equality.

An empirical frequency is an integer count divided by the number of observed trials. It may differ substantially from model probability at small sample sizes. The seeded sequence is reproducible in the reference environment, bounded, and illustrative; it is not a proof of a probability or a cross-language random-generator contract.

Engineering and ML connections

Finite models appear in categorical predictions, randomized tests, and simulation design. Before using a probability vector, check that labels are distinct, weights are nonnegative, and total mass is one. A model’s predicted probability and an observed class count answer different questions; mixing them can make evaluation circular.

Exercises

Enter exact fractions where appropriate. Each exercise offers corrective feedback and a worked solution after an attempt.

calculation

Count possible outcomes, then weight them

A fair six-sided die has A = {2, 4, 6}. Give |Ω| and P(A). Enter an exact fraction for the probability.

explanation

Keep the two coins distinguishable

Two independent fair coins are tossed in order. Which sample-space model is correct?

Choose one answer

calculation

Read weights instead of counting labels

A three-outcome experiment assigns masses 1/2, 1/3, 1/6 to red, green, blue. What is P({green, blue})?

prediction

Predict after a streak

Five observed rolls of a fair die were even. What does the specified independent fair-die model predict for an even sixth roll?

Choose one answer

explanation

Reject an invalid mass assignment

A proposed finite model has weights (1/2, 1/2, −1/4, 1/4), whose sum is 1. Is it valid?

Choose one answer

Summary and checkpoints

A sample space lists possibilities; a probability model weights them; an event selects a set; an observation records what happened. Check that you can justify the fair-die denominator, keep two coin positions distinct, sum unequal weights, and reject invalid models without confusing them with rare outcomes.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available