Mathematical Foundations

IV.3 · Probability

Does changing the order make a different outcome?

Choose labels with or without replacement, then decide whether order matters. Counts become probabilities only after you justify the weights of the counted outcomes.

Count the right kind of outcome

First 12 selections at most · labels A through H

  1. A → B
    1 ordering(s) · P = 1/12
  2. A → C
    1 ordering(s) · P = 1/12
  3. A → D
    1 ordering(s) · P = 1/12
  4. B → A
    1 ordering(s) · P = 1/12
  5. B → C
    1 ordering(s) · P = 1/12
  6. B → D
    1 ordering(s) · P = 1/12
  7. C → A
    1 ordering(s) · P = 1/12
  8. C → B
    1 ordering(s) · P = 1/12
  9. C → D
    1 ordering(s) · P = 1/12
  10. D → A
    1 ordering(s) · P = 1/12
  11. D → B
    1 ordering(s) · P = 1/12
  12. D → C
    1 ordering(s) · P = 1/12

4 labeled items, 2 draws, order matters, without replacement. 12 selections. Under uniform draws, these displayed selections are equiprobable.

Number of selections
12
Underlying ordered outcomes
12

= exact model quantity · ≈ numerical companion or observed estimate

from 2 to 8, whole numbers only

from 0 to 4, whole numbers only

Under uniform draws, these displayed selections are equiprobable.
N=n!(n−k)!=12N=\frac{n!}{(n-k)!}=12
P(S)=equiprobable ordered outcomes in Sall ordered outcomesP(S)=\frac{\text{equiprobable ordered outcomes in }S}{\text{all ordered outcomes}}

Specify what counts as one outcome

AB and BA are different; AA is forbidden.

n=4,k=2n=4,\quad k=2

Switch between ordered and unordered selections, toggle replacement, and change n from 2 to 8 and k from 0 to 4. Compare the outcomes and count before and after each change.

Begin with two selections from four labels without replacement. Predict the ordered count, then switch to unordered. Find the pair of ordered outcomes represented by one unordered selection. The stage names its labels A through H; the examples and Python use numbers 1 through 8 for the same distinct items.

Intuition: count what the experiment distinguishes

Giving a first prize to label 1 and a second prize to label 2 differs from giving the prizes in the reverse order. Choosing a two-person team from those labels ignores that order. The same labels therefore support different outcome spaces depending on the experiment’s question.

Replacement asks a separate question: can a label be used again? Keeping these two choices separate prevents a correct counting formula from being used for the wrong experiment.

Definition

Let nn be the number of distinct labels and kk the number selected, both integers. For ordered selection without replacement, there are

n(n−1)⋯(n−k+1)=n!(n−k)!n(n-1)\cdots(n-k+1)=\frac{n!}{(n-k)!}

outcomes when 0≤k≤n0\le k\le n. Ignoring order gives the combination count (nk)=n!/(k!(n−k)!)\binom nk=n!/(k!(n-k)!). The factorial convention is 0!=10!=1.

With replacement, ordered sequences number nkn^k. Unordered selections are multisets, which record how many times each label appears; for n≥1n\ge1 they number (n+k−1k)\binom{n+k-1}{k}. These are counts, not automatically a uniform probability model.

Mechanics: classify before calculating

The stage’s state records nn, kk, order, and replacement. Its outcome representation and count follow those choices. Read the derivation trace from the same state: identify available labels for each slot, multiply choices, and account for ignored order when appropriate.

The interactive range is 2≤n≤82\le n\le8, 0≤k≤40\le k\le4. It includes k>nk>n without replacement so you can see an impossible selection. Changing a control does not authorize an impossible draw or a probability with zero total outcomes. Reset returns to n=4n=4, k=2k=2, without replacement.

Derivation: multiplication and grouping

Without replacement, the first slot has nn choices, the next has n−1n-1, and so on. Multiply because each partial selection branches into that many next choices. Every unordered set of kk distinct labels has exactly k!k! orderings. Dividing the ordered count by k!k! therefore gives combinations.

With replacement, every slot has nn choices, giving nkn^k. To count unordered multisets, distribute kk indistinguishable marks among nn labelled boxes. A row of kk marks and n−1n-1 separators encodes the counts. Choosing the mark positions gives (n+k−1k)\binom{n+k-1}{k}.

Repeated labels make the number of orderings vary by multiset. If a multiset has counts c1,…,cnc_1,\ldots,c_n summing to kk, its number of ordered preimages is k!/(c1!⋯cn!)k!/(c_1!\cdots c_n!). Under independent uniform draws its probability is that number divided by nkn^k; this is why “ignore order” does not automatically mean “make all unordered results uniform.”

Worked examples

For n=4n=4, k=2k=2, without replacement, the ordered total is 4×3=124\times3=12. Each pair occurs in two orders, so there are 12/2=612/2=6 unordered pairs. The event “select labels 1 and 2” has probability 2/12=1/62/12=1/6 under uniform ordered draws; equivalently it is one of six equally likely unordered pairs.

Worked example: Count the right kind of outcome

First 12 selections at most · labels A through H

  1. A → B
    1 ordering(s) · P = 1/12
  2. A → C
    1 ordering(s) · P = 1/12
  3. A → D
    1 ordering(s) · P = 1/12
  4. B → A
    1 ordering(s) · P = 1/12
  5. B → C
    1 ordering(s) · P = 1/12
  6. B → D
    1 ordering(s) · P = 1/12
  7. C → A
    1 ordering(s) · P = 1/12
  8. C → B
    1 ordering(s) · P = 1/12
  9. C → D
    1 ordering(s) · P = 1/12
  10. D → A
    1 ordering(s) · P = 1/12
  11. D → B
    1 ordering(s) · P = 1/12
  12. D → C
    1 ordering(s) · P = 1/12

4 labeled items, 2 draws, order matters, without replacement. 12 selections. Under uniform draws, these displayed selections are equiprobable.

Number of selections
12
Underlying ordered outcomes
12

= exact model quantity · ≈ numerical companion or observed estimate

from 2 to 8, whole numbers only

from 0 to 4, whole numbers only

Under uniform draws, these displayed selections are equiprobable.
N=n!(n−k)!=12N=\frac{n!}{(n-k)!}=12
P(S)=equiprobable ordered outcomes in Sall ordered outcomesP(S)=\frac{\text{equiprobable ordered outcomes in }S}{\text{all ordered outcomes}}

Specify what counts as one outcome

AB and BA are different; AA is forbidden.

n=4,k=2n=4,\quad k=2

Change the worked two-of-four selection from ordered to unordered and inspect the 12-to-6 grouping. Turn replacement on to compare 16 sequences with 10 multisets.

With replacement, there are 42=164^2=16 sequences and (52)=10\binom52=10 multisets. Four multisets repeat a label; six contain distinct labels. The result {1,1}\{1,1\} has one preimage and mass 1/161/16. The result {1,2}\{1,2\} has two preimages and mass 1/81/8. Assigning 1/101/10 to each describes a different experiment: direct uniform selection of multisets.

Executable reference code

The Python program uses standard-library formulas and independently enumerates outcomes with itertools to check them throughout the stage’s small parameter range. It groups replacement sequences by sorted tuples and assigns exact Fraction masses from preimage counts. NumPy checks the same finite grouping using rows of sequences, row sorting, and occurrence counts; no random sampling is needed.

code/python/probability_counting.py
"""Count outcomes, then justify their masses; never assume multisets are uniform.

Enumeration is bounded to the stage's n = 2..8, k = 0..4. Formulas return exact
integers; probabilities below use Fraction, not floating-point count ratios.
"""
from collections import Counter
from fractions import Fraction as F
from itertools import combinations, combinations_with_replacement, permutations, product
from math import comb, perm


def validate(n, k):
    if type(n) is not int or type(k) is not int or not 2 <= n <= 8 or not 0 <= k <= 4:
        raise ValueError("This explorer requires integer n = 2..8 and k = 0..4")


def selection_count(n, k, ordered, replacement):
    validate(n, k)
    if replacement:
        return n ** k if ordered else comb(n + k - 1, k)
    if k > n:
        return 0
    return perm(n, k) if ordered else comb(n, k)


def enumerate_selections(n, k, ordered, replacement):
    validate(n, k)
    labels = range(1, n + 1)
    if replacement:
        return tuple(product(labels, repeat=k) if ordered else combinations_with_replacement(labels, k))
    return tuple(permutations(labels, k) if ordered else combinations(labels, k))


def multiset_masses(n, k):
    sequences = enumerate_selections(n, k, ordered=True, replacement=True)
    preimages = Counter(tuple(sorted(row)) for row in sequences)
    return {multiset: F(count, len(sequences)) for multiset, count in preimages.items()}


def main():
    assert selection_count(4, 2, True, False) == 12
    assert selection_count(4, 2, False, False) == 6
    assert selection_count(4, 2, True, True) == 16
    assert selection_count(4, 2, False, True) == 10
    assert selection_count(2, 3, True, False) == 0
    assert selection_count(2, 3, False, False) == 0
    for n in range(2, 9):
        for k in range(5):
            for ordered, replacement in product((False, True), repeat=2):
                outcomes = enumerate_selections(n, k, ordered, replacement)
                assert len(outcomes) == selection_count(n, k, ordered, replacement)
                if k == 0:
                    assert outcomes == ((),)
            masses = multiset_masses(n, k)
            assert sum(masses.values(), F(0)) == 1
    for n, k in ((1, 2), (9, 2), (4, -1), (4, 5), (4.0, 2), (4, True)):
        try:
            selection_count(n, k, True, False)
        except ValueError:
            pass
        else:
            raise AssertionError("Out-of-range or noninteger input was accepted")
    pairs = enumerate_selections(4, 2, True, False)
    target_mass = F(sum(set(row) == {1, 2} for row in pairs), len(pairs))
    assert target_mass == F(1, 6)
    masses = multiset_masses(4, 2)
    assert masses[(1, 1)] == F(1, 16)
    assert masses[(1, 2)] == F(1, 8)
    print("Two of four, without replacement: ordered 12, unordered 6; P({1,2}) =", target_mass)
    print("With replacement: ordered 16, unordered 10; exact induced multiset masses:", masses)
    print("k = 0: one empty outcome; k > n without replacement: no sampling space")


if __name__ == "__main__":
    main()
code/python/probability_counting_numpy.py
"""Group bounded ordered selections with NumPy, keeping multiplicities visible.

Counts in this n = 2..8, k = 0..4 explorer fit fixed-width integers. Derived
floating-point probabilities use absolute tolerance 1e-12, relative tolerance 0.
"""
from fractions import Fraction as F
from itertools import permutations, product
from math import comb, perm
import numpy as np

ATOL = 1e-12


def grouped_selections(n, k, replacement):
    if type(n) is not int or type(k) is not int or not 2 <= n <= 8 or not 0 <= k <= 4:
        raise ValueError("Use integer n = 2..8 and k = 0..4")
    labels = range(1, n + 1)
    sequences = tuple(product(labels, repeat=k) if replacement else permutations(labels, k))
    rows = np.asarray(sequences, dtype=np.int64).reshape(len(sequences), k)
    if len(rows) == 0:
        return rows, np.empty((0, k), dtype=np.int64), np.empty(0, dtype=np.int64), None
    # Sorting removes order; counts retain how many ordered rows each result had.
    unordered, multiplicities = np.unique(np.sort(rows, axis=1), axis=0, return_counts=True)
    masses = multiplicities / len(rows)
    return rows, unordered, multiplicities, masses


def main():
    for n in range(2, 9):
        for k in range(5):
            for replacement in (False, True):
                rows, unordered, counts, masses = grouped_selections(n, k, replacement)
                ordered_count = n ** k if replacement else (perm(n, k) if k <= n else 0)
                unordered_count = comb(n + k - 1, k) if replacement else (comb(n, k) if k <= n else 0)
                assert len(rows) == ordered_count and len(unordered) == unordered_count
                if masses is not None:
                    assert counts.sum() == ordered_count
                    np.testing.assert_allclose(masses.sum(), 1.0, rtol=0, atol=ATOL)
                if k == 0:
                    assert rows.shape == unordered.shape == (1, 0) and counts.tolist() == [1]
    rows, pairs, counts, masses = grouped_selections(4, 2, False)
    assert len(rows) == 12 and len(pairs) == 6 and np.all(counts == 2)
    np.testing.assert_allclose(masses, float(F(1, 6)), rtol=0, atol=ATOL)
    rows_r, multisets, counts_r, masses_r = grouped_selections(4, 2, True)
    assert len(rows_r) == 16 and len(multisets) == 10
    repeated = np.all(multisets == [1, 1], axis=1)
    distinct = np.all(multisets == [1, 2], axis=1)
    np.testing.assert_allclose(masses_r[repeated], float(F(1, 16)), rtol=0, atol=ATOL)
    np.testing.assert_allclose(masses_r[distinct], float(F(1, 8)), rtol=0, atol=ATOL)
    assert grouped_selections(2, 3, False)[3] is None
    for n, k in ((1, 2), (9, 2), (4, -1), (4, 5), (4.0, 2)):
        try:
            grouped_selections(n, k, False)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid explorer inputs were accepted")
    print("Without replacement: 12 sequences, 6 pairs; probabilities ≈", masses)
    print("With replacement: 16 sequences, 10 multisets")
    for row, count, mass in zip(multisets, counts_r, masses_r):
        print(tuple(row), ": ordered preimages", count, "; probability ≈", mass)
    print("Impossible without-replacement selection: no normalized masses (None)")


if __name__ == "__main__":
    main()

Explore

Set k=0k=0 in all four modes. There is exactly one empty selection, including one empty ordered tuple. Under this valid experiment its probability is one. Then set n=2n=2, k=3k=3 without replacement: there are zero selections, so a normalized uniform selection model cannot be formed. Turn replacement on and observe that ordered choices become possible again.

Predict counts for n=5n=5, k=3k=3 without replacement: 5×4×3=605\times4\times3=60 ordered and 60/6=1060/6=10 unordered. Return to two-of-four with replacement and inspect which unordered results represent more sequences.

Common mistakes and counterexamples

Dividing every ordered replacement count by k!k! fails when labels repeat. For two-of-four it gives 16/2=816/2=8, but there are ten multisets. Using nkn^k without replacement also overcounts repeated-label sequences that the experiment excludes.

An impossible selection is not an empty selection. One has no outcome; the other has the single empty outcome. Do not evaluate a factorial with a negative argument to extend the without-replacement formula past k=nk=n; report count zero and the unavailable sampling model.

Numerical considerations

Counts are exact integers and probabilities are exact rational ratios in the standard-library reference. The bounded explorer permits enumeration, but nkn^k grows quickly outside that range. A finite display must never be mistaken for an exhaustive list in a much larger experiment.

Python integers grow as needed. NumPy integer arrays have fixed-width types, so a larger counting implementation must consider overflow before arithmetic. Here the bounded counts fit exactly; division into floating-point masses is approximate, checked with absolute tolerance 10−1210^{-12} and zero relative tolerance. The reference does not claim a rounded array sum is an exact proof of normalization.

Engineering and ML connections

Sampling a minibatch without replacement differs from independently choosing training examples with replacement. Sequence order matters for temporal input, whereas a selected subset may ignore it. Counting helps estimate search-space size and memory requirements, but probabilities still depend on the sampling rule. Grouping randomized outputs can introduce unequal category probabilities even when the underlying draws are uniform.

Exercises

calculation

Count two selection models

Choose two different labels from {1, 2, 3, 4}, without replacement. Give the ordered and unordered counts.

calculation

Allow repeats and inspect the quotient

Select two labels from four with replacement. Give the ordered sequence count and the distinct unordered multiset count.

calculation

Put the favorable count over the right total

Draw two different labels uniformly without replacement from {1, 2, 3, 4}. Ignoring order, what is the probability of getting exactly {1, 2}?

explanation

Do all unordered results have equal mass?

Two independent uniform draws from {1, 2, 3, 4} allow replacement; then order is hidden. Compare the multisets {1, 1} and {1, 2}.

Choose one answer

explanation

Handle the boundary cases

What are the selection counts for k = 0, and for n = 2, k = 3 without replacement?

Choose one answer

Summary and checkpoints

Choose an outcome space by specifying order and replacement. Multiply successive choices; divide by equal-size ordering groups only when justified. Check that you can explain 12 versus 6, 16 versus 10, the nonuniform replacement multisets, and the difference between one empty outcome and no possible outcome.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available