Mathematical Foundations

IV.2 · Probability

How much mass belongs to either event?

Toggle membership in two events and follow union, intersection, and complement. Overlap is counted once, and the underlying outcome weights stay the same.

Events carry probability mass

Outlined outcomes belong to A ∪ B.

1Original P = 1/6A ∪ B: excluded
2Original P = 1/6A ∪ B: included
3Original P = 1/6A ∪ B: excluded
4Original P = 1/6A ∪ B: included
5Original P = 1/6A ∪ B: included
6Original P = 1/6A ∪ B: included
1= 1/6
2= 1/6
3= 1/6
4= 1/6
5= 1/6
6= 1/6

Sample space: 1, 2, 3, 4, 5, 6. Equal weights explicitly assume equiprobable outcomes. A = {2, 4, 6}; B = {4, 5, 6}. Union includes either event; intersection includes both; complement includes outcomes outside A. Every mass uses the same weights.

Total probability
1
Event A
0.5
Event B
0.5
Intersection
1/30.333333…
Union
2/30.666667…
Complement of A
0.5

= exact model quantity · ≈ numerical companion or observed estimate

from 0 to 100, whole numbers only

Union includes either event; intersection includes both; complement includes outcomes outside A. Every mass uses the same weights.
P(A)=∑ω∈Apω=0.5P(A)=\sum_{\omega\in A}p_\omega=0.5
P(A∪B)=P(A)+P(B)−P(A∩B)=23P(A\cup B)=P(A)+P(B)-P(A\cap B)=\frac{2}{3}

Start with total mass one

Equal weights explicitly assume equiprobable outcomes.

∑ωpω=1\sum_\omega p_\omega=1
1= 1/6
2= 1/6
3= 1/6
4= 1/6
5= 1/6
6= 1/6

Select outcomes for A and B. Move a face into both events, then out of one; compare intersection and union mass as the membership labels and formulas update together.

Start with the fair die’s A={2,4,6}A=\{2,4,6\} and B={4,5,6}B=\{4,5,6\}. Predict whether adding the two half-masses gives the union’s mass. Locate the two faces that would be counted twice.

Intuition: mass lives on outcomes

Events do not own separate copies of probability. They select from a single space whose weights sum to one. An outcome that belongs to both events still has one weight. This is why overlapping regions cannot simply be added when the question is “A or B.”

The geometry of overlap is set membership. Probability measures its mass. Two regions with the same number of labels can have different probabilities when their weights differ.

Definition

For events A,B⊆ΩA,B\subseteq\Omega, the union A∪BA\cup B contains outcomes in at least one event; the intersection A∩BA\cap B contains outcomes in both. The complement Ac=Ω∖AA^c=\Omega\setminus A contains outcomes outside AA but inside the chosen sample space. Events are disjoint when A∩B=∅A\cap B=\varnothing.

For every event, P(A)=∑ω∈ApωP(A)=\sum_{\omega\in A}p_\omega, using the same nonnegative, normalized weights. Therefore 0≤P(A)≤10\le P(A)\le1, P(∅)=0P(\varnothing)=0, and P(Ω)=1P(\Omega)=1.

Mechanics: follow membership and weight separately

The stage uses outcome membership to show the two events and their overlap. Toggling A or B changes a set, not its probability weight. Edit weight selects a nonnegative raw weight to change; all raw weights are divided by their positive total to obtain one normalized mass model. Read the labels along with the shading: colour alone should not be your evidence that a face belongs to both events.

Inspect the original event masses and the combined sets. Replay the trace after moving a face from A-only to both: its intersection contribution appears, but its union contribution was already present. The inspector, equation, and accessible interpretation describe the same memberships and weights.

Derivation: inclusion-exclusion

Separate the union into three disjoint pieces: A∖BA\setminus B, A∩BA\cap B, and B∖AB\setminus A. Adding P(A)+P(B)P(A)+P(B) includes the middle piece twice. Subtract it once:

P(A∪B)=P(A)+P(B)−P(A∩B).P(A\cup B)=P(A)+P(B)-P(A\cap B).

The sample space also splits into the disjoint pieces AA and AcA^c, so

P(Ac)=1−P(A).P(A^c)=1-P(A).

These identities require neither equal outcome weights nor independence. For disjoint events the intersection is empty, and the subtraction term is zero. For A=BA=B, the union is simply AA, not twice its mass.

Worked examples

On the fair die, P(A)=P(B)=1/2P(A)=P(B)=1/2. The intersection {4,6}\{4,6\} has mass 1/31/3. The union {2,4,5,6}\{2,4,5,6\} has mass 2/32/3, agreeing with 1/2+1/2−1/31/2+1/2-1/3. The complement Ac={1,3,5}A^c=\{1,3,5\} has mass 1/21/2.

Worked example: Events carry probability mass

Outlined outcomes belong to A ∪ B.

1Original P = 1/6A ∪ B: excluded
2Original P = 1/6A ∪ B: included
3Original P = 1/6A ∪ B: excluded
4Original P = 1/6A ∪ B: included
5Original P = 1/6A ∪ B: included
6Original P = 1/6A ∪ B: included
1= 1/6
2= 1/6
3= 1/6
4= 1/6
5= 1/6
6= 1/6

Sample space: 1, 2, 3, 4, 5, 6. Equal weights explicitly assume equiprobable outcomes. A = {2, 4, 6}; B = {4, 5, 6}. Union includes either event; intersection includes both; complement includes outcomes outside A. Every mass uses the same weights.

Total probability
1
Event A
0.5
Event B
0.5
Intersection
1/30.333333…
Union
2/30.666667…
Complement of A
0.5

= exact model quantity · ≈ numerical companion or observed estimate

from 0 to 100, whole numbers only

Union includes either event; intersection includes both; complement includes outcomes outside A. Every mass uses the same weights.
P(A)=∑ω∈Apω=0.5P(A)=\sum_{\omega\in A}p_\omega=0.5
P(A∪B)=P(A)+P(B)−P(A∩B)=23P(A\cup B)=P(A)+P(B)-P(A\cap B)=\frac{2}{3}

Start with total mass one

Equal weights explicitly assume equiprobable outcomes.

∑ωpω=1\sum_\omega p_\omega=1
1= 1/6
2= 1/6
3= 1/6
4= 1/6
5= 1/6
6= 1/6

Toggle face 4 out of B, then restore it. Follow how removing a shared 1/6 contribution changes the intersection while leaving that face in the union through A.

For a weighted space Ω={x,y,z}\Omega=\{x,y,z\} with masses 1/2,1/3,1/61/2,1/3,1/6, choose A={x,y}A=\{x,y\} and B={y,z}B=\{y,z\}. The marginal masses are 5/65/6 and 1/21/2, intersection mass is 1/31/3, and union mass is 11. Inclusion-exclusion still gives 5/6+1/2−1/3=15/6+1/2-1/3=1.

Executable reference code

The Python program validates the model, performs actual set operations, and sums exact fractions over each result. Assertions check the literal die answers and weighted identities. NumPy represents membership with boolean masks: OR for union, AND for intersection, and NOT for complement. Its printed decimal sums approximate the exact reference fractions.

code/python/probability_events.py
"""Set operations select outcomes; one normalized mass map weights every set."""
from fractions import Fraction as F


def event_mass(weights, event):
    if not weights or any(not isinstance(p, F) or p < 0 for p in weights.values()):
        raise ValueError("Use a nonempty model with nonnegative Fraction masses")
    if sum(weights.values(), F(0)) != 1:
        raise ValueError("Masses must sum exactly to one")
    if not set(event) <= weights.keys():
        raise ValueError("Event contains an outcome outside the sample space")
    return sum((weights[x] for x in set(event)), F(0))


def describe_events(weights, a, b):
    a, b = set(a), set(b)
    # Validate even events whose out-of-space labels disappear in an intersection.
    pa, pb = event_mass(weights, a), event_mass(weights, b)
    joint = event_mass(weights, a & b)
    union = event_mass(weights, a | b)
    complement = event_mass(weights, set(weights) - a)
    assert union == pa + pb - joint
    assert complement == 1 - pa
    return {"P(A)": pa, "P(B)": pb, "P(A∩B)": joint,
            "P(A∪B)": union, "P(Aᶜ)": complement, "disjoint": not (a & b)}


def main():
    die = {face: F(1, 6) for face in range(1, 7)}
    a, b = {2, 4, 6}, {4, 5, 6}
    result = describe_events(die, a, b)
    assert result == {"P(A)": F(1, 2), "P(B)": F(1, 2), "P(A∩B)": F(1, 3),
                      "P(A∪B)": F(2, 3), "P(Aᶜ)": F(1, 2), "disjoint": False}
    assert describe_events(die, a, a)["P(A∪B)"] == F(1, 2)
    assert describe_events(die, a, set(die) - a)["P(A∪B)"] == 1
    assert describe_events(die, set(), b)["P(A∩B)"] == 0
    weighted = {"x": F(1, 2), "y": F(1, 3), "z": F(1, 6)}
    weighted_result = describe_events(weighted, {"x", "y"}, {"y", "z"})
    assert weighted_result["P(A∩B)"] == F(1, 3)
    assert weighted_result["P(A∪B)"] == 1
    zero_weight = {"possible": F(1), "zero": F(0)}
    overlap = describe_events(zero_weight, {"zero"}, {"zero"})
    assert overlap["P(A∩B)"] == 0 and not overlap["disjoint"]
    for weights, event in ((die, {7}), ({}, set()),
                           ({1: F(-1), 2: F(2)}, {1}), ({1: F(1, 2)}, {1})):
        try:
            describe_events(weights, event, set())
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid model or event was accepted")
    print("Fair die, A = {2, 4, 6}, B = {4, 5, 6}:", result)
    print("Weighted x, y, z example:", weighted_result)
    print("Shared zero-weight label: joint mass 0, but sets are not disjoint")


if __name__ == "__main__":
    main()
code/python/probability_events_numpy.py
"""OR, AND, and NOT implement set membership; weights supply event mass.

Membership is boolean and exact. Floating-point sums are compared to literal
Fraction references at absolute tolerance 1e-12, with zero relative tolerance.
"""
from fractions import Fraction as F
import numpy as np

ATOL = 1e-12


def event_summary(weights, a, b):
    p = np.asarray(weights, dtype=float)
    a, b = np.asarray(a), np.asarray(b)
    if p.ndim != 1 or p.size == 0 or np.any(p < 0) or not np.all(np.isfinite(p)):
        raise ValueError("Use a nonempty finite nonnegative mass vector")
    if not np.isclose(p.sum(), 1.0, rtol=0, atol=ATOL):
        raise ValueError("Masses must sum to one within floating-point tolerance")
    if a.dtype != np.bool_ or b.dtype != np.bool_ or a.shape != p.shape or b.shape != p.shape:
        raise ValueError("Use boolean event masks with one entry per outcome")
    pa, pb = p[a].sum(), p[b].sum()
    joint, union, complement = p[a & b].sum(), p[a | b].sum(), p[~a].sum()
    np.testing.assert_allclose(union, pa + pb - joint, rtol=0, atol=ATOL)
    np.testing.assert_allclose(complement, 1 - pa, rtol=0, atol=ATOL)
    return np.array([pa, pb, joint, union, complement]), not np.any(a & b)


def main():
    faces = np.arange(1, 7)
    p = np.full(6, 1 / 6)
    a, b = np.isin(faces, [2, 4, 6]), np.isin(faces, [4, 5, 6])
    masses, disjoint = event_summary(p, a, b)
    expected = [F(1, 2), F(1, 2), F(1, 3), F(2, 3), F(1, 2)]
    np.testing.assert_allclose(masses, [float(x) for x in expected], rtol=0, atol=ATOL)
    assert not disjoint
    assert event_summary(p, a, ~a)[1]
    np.testing.assert_allclose(event_summary(p, a, a)[0][3], float(F(1, 2)), rtol=0, atol=ATOL)
    weighted, _ = event_summary([0.5, 1 / 3, 1 / 6], [True, True, False], [False, True, True])
    np.testing.assert_allclose(weighted, [float(x) for x in (F(5, 6), F(1, 2), F(1, 3), F(1), F(1, 6))], rtol=0, atol=ATOL)
    zero_overlap, zero_disjoint = event_summary([1.0, 0.0], [False, True], [False, True])
    assert zero_overlap[2] == 0 and not zero_disjoint
    for bad_p, bad_a, bad_b in (([-0.5, 1.5], [True, False], [False, True]),
                                 ([0.5, 0.4], [True, False], [False, True]),
                                 ([np.nan], [True], [True]),
                                 ([1.0], [1], [True]), ([1.0], [True, False], [True])):
        try:
            event_summary(bad_p, bad_a, bad_b)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid masses or masks were accepted")
    print("Die masses ≈", masses, "in order A, B, intersection, union, complement")
    print("Exact references:", expected)
    print("Weighted example masses ≈", weighted)
    print("Zero overlap mass does not guarantee disjoint sets")


if __name__ == "__main__":
    main()

Explore

Make B equal to A. Predict that intersection and union both become A. Then choose B to be the complement of A: the intersection becomes empty and the union becomes the whole space. Finally empty A while keeping B fixed; explain why the union remains B and the intersection has mass zero.

For every change, check inclusion-exclusion against direct union summation. A trace that adds weights over selected outcomes provides a second view of the same mathematical identity, rather than evidence from random trials.

Common mistakes and counterexamples

“Or” in union is inclusive: an outcome in both counts. Exclusive OR would omit the intersection and describe a different event. A complement is relative to the declared space; the complement of even die faces does not include the integer 77.

Equal probabilities do not imply equal sets. The even faces and the top three faces each have mass 1/21/2 but differ. Disjointness also does not mean independence: odd and even are mutually exclusive and each has positive mass, so knowing one happened rules out the other.

Numerical considerations

Rational summation preserves exact union and complement identities for these finite examples. Floating-point code may accumulate tiny summation errors. The NumPy reference checks its small arrays against exact fractions with absolute tolerance 10−1210^{-12} and zero relative tolerance; it does not silently clip a negative weight or renormalize an invalid vector.

Boolean set membership itself is exact. Keep that separate from approximate mass arithmetic. A tiny rounded joint mass should not decide whether two sets are disjoint, and a displayed decimal such as 0.33330.3333 is not the exact fraction 1/31/3.

Engineering and ML connections

Boolean masks select rows, category groups, and evaluation subsets. A union of error categories requires overlap correction or an OR mask; adding category counts can double-count the same examples. Weighted data requires summing weights over the mask. The same structure appears when grouping predicted categorical probabilities.

Exercises

calculation

Count overlap once

For a fair die, A = {2, 4, 6}, B = {4, 5, 6}. Give P(A ∩ B) and P(A ∪ B).

calculation

Stay inside the sample space

On the same fair die, find P(Aᶜ) for A = {2, 4, 6}.

calculation

Use set membership with unequal masses

Outcomes x, y, z have masses 1/2, 1/3, 1/6. A = {x, y}, B = {y, z}. Give P(A ∩ B) and P(A ∪ B).

explanation

Distinguish disjointness from independence

On a fair die, C = {1, 3, 5} and D = {2, 4, 6}. What is true?

Choose one answer

explanation

Choose the correct mask operation

NumPy boolean arrays a and b mark event membership; p holds outcome masses. Which expression computes union mass?

Choose one answer

Summary and checkpoints

Events select outcomes from one weighted space. Union counts each selected outcome once; intersection requires both memberships; complement fills the rest of the space. Check that you can derive inclusion-exclusion, apply it to unequal weights, and distinguish set disjointness from zero mass and independence.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available