Mathematical Foundations

IV.9 · Probability

How many different sequences produce the same success count?

Begin with one binary trial, then group independent trial sequences by their total successes. The binomial coefficient counts the ways to place those successes.

Count successes in independent trials

PMF · probability at a numerical value
PMF · probability at a numerical valueBinomial n=8, p=35 percent. Selected k=3, exact mass 0.2785857790625. Selected closed interval [0, 8]. PMF bars carry exact point masses. The closed interval has mass 1. A CDF includes the jump at its argument: subtract F(ℓ−), not F(ℓ), for a closed interval.10: approximately 0.03186448128906251: approximately 0.13726238093752: approximately 0.258686794843753: approximately 0.27858577906254: approximately 0.1875096589843755: approximately 0.08077339156256: approximately 0.021746682343757: approximately 0.00334564343758: approximately 0.000225187539062502468

Success count k · colored selected bars carry probability mass

CDF · nondecreasing and continuous from the right
CDF · nondecreasing and continuous from the rightBinomial n=8, p=35 percent. Selected k=3, exact mass 0.2785857790625. Selected closed interval [0, 8]. PMF bars carry exact point masses. The closed interval has mass 1. A CDF includes the jump at its argument: subtract F(ℓ−), not F(ℓ), for a closed interval.102468

Filled dot gives F(k); an open dot marks the left limit at a positive jump. F ends at one.

Binomial n=8, p=35 percent. Selected k=3, exact mass 0.2785857790625. Selected closed interval [0, 8]. PMF bars carry exact point masses. The closed interval has mass 1. A CDF includes the jump at its argument: subtract F(ℓ−), not F(ℓ), for a closed interval.

Selected binomial mass
0.2785857790625
Total finite mass
1
Expectation
2.8
Variance
1.82

= exact model quantity · ≈ numerical companion or observed estimate

from 0 to 30, whole numbers only

from 0 to 100, up to 2 decimal places

from −1 to 31, whole numbers only

PMF bars carry exact point masses. The closed interval has mass 1. A CDF includes the jump at its argument: subtract F(ℓ−), not F(ℓ), for a closed interval.
P(X=k)=(nk)pk(1−p)n−kp=0.35P(X=k)=\binom nk p^k(1-p)^{n-k}\quad p=0.35
P(X=3)=0.2785857790625P(X=3)=0.2785857790625
∑k=0n(nk)pk(1−p)n−k=(p+1−p)n=1\sum_{k=0}^n\binom nk p^k(1-p)^{n-k}=(p+1-p)^n=1

Start with individual trial outcomes

8 iid Bernoulli trials, p=0.35. Empty products are one in this counting context; n=0 means zero successes with probability one.

P(X=k)=(nk)pk(1−p)n−kp=0.35P(X=k)=\binom nk p^k(1-p)^{n-k}\quad p=0.35

Change trial count n and success probability p, then select a count k. Set n = 2 and p = 50%; the middle count has twice the mass of either extreme.

Predict the three masses before reading them. Set the selected count outside the support and distinguish an impossible event from an invalid experiment.

Intuition: binary trials, numerical totals

A trial records success as 11 and failure as 00. “Success” is simply the event being counted; it need not mean a desirable result. One trial is a Bernoulli variable. Adding several trials produces a success count.

For two fair independent trials, the sequences are 00,01,10,1100,01,10,11. Their counts are 0,1,1,20,1,1,2. The same grouping principle from random variables yields three count values, with more mass in the middle because two sequences feed it.

Definition

A Bernoulli variable BB with parameter p∈[0,1]p\in[0,1] has P(B=1)=pP(B=1)=p and P(B=0)=1−pP(B=0)=1-p. If B1,…,BnB_1,\ldots,B_n are independent Bernoulli variables with the same pp, then

X=∑i=1nBi∼Binomial⁡(n,p),P(X=k)=(nk)pk(1−p)n−k,0≤k≤n.X=\sum_{i=1}^n B_i\sim\operatorname{Binomial}(n,p), \qquad P(X=k)=\binom nk p^k(1-p)^{n-k},\quad 0\le k\le n.

Outside that integer support, the mass is zero. The parameter nn is a nonnegative integer; n=0n=0 is valid. For n=1n=1, the binomial model is Bernoulli. Independence and a common success probability are assumptions, not consequences of using binary labels.

Mechanics: parameters and selected events

The stage accepts integer nn from 00 through 3030, success percentage from 00 through 100100 with two decimal places, and selected integer kk from −1-1 through 3131. These are teaching controls; divide a percentage by 100100 to obtain the probability parameter. The selected kk identifies one event without changing the distribution.

Follow the trace from selecting kk success positions, through the probability of one such sequence, to their product. Compare the selected PMF mass with its CDF jump. Selecting k=−1k=-1 or k>nk>n yields mass zero; it must not be silently clamped to a possible count. Invalid parameters, such as negative nn or p>1p>1, instead fail to define this model.

Derivation: multiply first, then count

Independence makes any particular length-nn sequence with kk successes have mass pk(1−p)n−kp^k(1-p)^{n-k}. A common pp makes this mass the same for every sequence with that count. There are (nk)\binom nk choices of success positions, giving the PMF above.

The binomial theorem checks its total:

∑k=0n(nk)pk(1−p)n−k=(p+(1−p))n=1.\sum_{k=0}^n\binom nk p^k(1-p)^{n-k} =(p+(1-p))^n=1.

With zero trials, the one empty sequence has success count 00 and mass 11. A zero exponent here denotes an empty product of value 11, including boundary factors written 000^0 in this combinatorial formula. This convention concerns empty products; it is not a universal statement about indeterminate limits.

Worked examples

For n=2n=2, p=1/2p=1/2, all four sequences have mass 1/41/4. Grouping gives P(X=0)=1/4P(X=0)=1/4, P(X=1)=1/2P(X=1)=1/2, P(X=2)=1/4P(X=2)=1/4. A single count label does not mean a single source sequence.

Worked example: Count successes in independent trials

PMF · probability at a numerical value
PMF · probability at a numerical valueBinomial n=8, p=35 percent. Selected k=3, exact mass 0.2785857790625. Selected closed interval [0, 8]. PMF bars carry exact point masses. The closed interval has mass 1. A CDF includes the jump at its argument: subtract F(ℓ−), not F(ℓ), for a closed interval.10: approximately 0.03186448128906251: approximately 0.13726238093752: approximately 0.258686794843753: approximately 0.27858577906254: approximately 0.1875096589843755: approximately 0.08077339156256: approximately 0.021746682343757: approximately 0.00334564343758: approximately 0.000225187539062502468

Success count k · colored selected bars carry probability mass

CDF · nondecreasing and continuous from the right
CDF · nondecreasing and continuous from the rightBinomial n=8, p=35 percent. Selected k=3, exact mass 0.2785857790625. Selected closed interval [0, 8]. PMF bars carry exact point masses. The closed interval has mass 1. A CDF includes the jump at its argument: subtract F(ℓ−), not F(ℓ), for a closed interval.102468

Filled dot gives F(k); an open dot marks the left limit at a positive jump. F ends at one.

Binomial n=8, p=35 percent. Selected k=3, exact mass 0.2785857790625. Selected closed interval [0, 8]. PMF bars carry exact point masses. The closed interval has mass 1. A CDF includes the jump at its argument: subtract F(ℓ−), not F(ℓ), for a closed interval.

Selected binomial mass
0.2785857790625
Total finite mass
1
Expectation
2.8
Variance
1.82

= exact model quantity · ≈ numerical companion or observed estimate

from 0 to 30, whole numbers only

from 0 to 100, up to 2 decimal places

from −1 to 31, whole numbers only

PMF bars carry exact point masses. The closed interval has mass 1. A CDF includes the jump at its argument: subtract F(ℓ−), not F(ℓ), for a closed interval.
P(X=k)=(nk)pk(1−p)n−kp=0.35P(X=k)=\binom nk p^k(1-p)^{n-k}\quad p=0.35
P(X=3)=0.2785857790625P(X=3)=0.2785857790625
∑k=0n(nk)pk(1−p)n−k=(p+1−p)n=1\sum_{k=0}^n\binom nk p^k(1-p)^{n-k}=(p+1-p)^n=1

Start with individual trial outcomes

8 iid Bernoulli trials, p=0.35. Empty products are one in this counting context; n=0 means zero successes with probability one.

P(X=k)=(nk)pk(1−p)n−kp=0.35P(X=k)=\binom nk p^k(1-p)^{n-k}\quad p=0.35

Recreate the two-trial PMF, then try n = 0, p = 0%, and p = 100%. The distribution remains valid in each boundary case.

For n=3n=3, p=1/4p=1/4, exactly one success can occupy three positions. Each corresponding sequence has mass (1/4)(3/4)2=9/64(1/4)(3/4)^2=9/64, so

P(X=1)=3×964=2764.P(X=1)=3\times\frac9{64}=\frac{27}{64}.

At p=0p=0, all mass lies at k=0k=0. At p=1p=1, all mass lies at k=nk=n. For n=0n=0, these both name the same count. No factorial of a negative integer is needed for an impossible selected event.

Executable reference code

The exact program uses standard-library comb and Fraction, returning zero before evaluating an out-of-support count. It checks literal worked values and sweeps n=0,…,30n=0,\ldots,30 across deterministic, near-boundary, and ordinary probabilities, asserting normalization and mean npnp. Fraction('12.34') / 100 represents 12.34%12.34\% exactly.

The NumPy program exposes the coefficient array and elementwise powers. It compares each mass against exact rational products within absolute tolerance 10−1210^{-12}, with no SciPy dependency and no random sampling.

code/python/probability_binomial.py
"""Exact binomial masses; no factorial at an impossible count.

The stage accepts n in 0..30, selected k in -1..31, and p-percent to two
decimal places. Fraction('12.34') / 100 represents 12.34% exactly.
"""
from fractions import Fraction as F
from math import comb


def pmf(n, p, k):
    if type(n) is not int or not 0 <= n <= 30 or type(k) is not int or not -1 <= k <= 31:
        raise ValueError("Require integer n in 0..30 and integer k in -1..31")
    if not isinstance(p, F) or not 0 <= p <= 1:
        raise ValueError("Success probability must be a Fraction in [0, 1]")
    if k < 0 or k > n:
        return F(0)
    # An exponent of zero denotes the empty product, including at p = 0 or 1.
    return comb(n, k) * p**k * (1 - p)**(n - k)


def main():
    assert [pmf(2, F(1, 2), k) for k in range(3)] == [F(1, 4), F(1, 2), F(1, 4)]
    assert pmf(3, F(1, 4), 1) == F(27, 64)
    assert pmf(0, F(2, 5), 0) == 1
    assert pmf(0, F(2, 5), 1) == 0
    assert pmf(3, F(0), 0) == pmf(3, F(1), 3) == 1
    assert pmf(2, F(1, 2), -1) == pmf(2, F(1, 2), 3) == 0
    percentage = F('12.34') / 100
    assert percentage == F(617, 5000)
    for n in range(31):
        for p in (F(0), F(1, 10000), percentage, F(1, 2), F(9999, 10000), F(1)):
            distribution = [pmf(n, p, k) for k in range(n + 1)]
            assert sum(distribution, F(0)) == 1
            assert sum((k * mass for k, mass in enumerate(distribution)), F(0)) == n * p
    for args in ((-1, F(1, 2), 0), (31, F(1, 2), 0), (2, F(-1), 0), (2, F(2), 0), (2, 0.5, 1), (2, F(1, 2), 1.5)):
        try:
            pmf(*args)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid binomial parameters accepted")
    print("Binomial(2, 1/2) exact:", [pmf(2, F(1, 2), k) for k in range(3)])
    print("Binomial(3, 1/4), k = 1:", pmf(3, F(1, 4), 1))
    print("Zero trials:", pmf(0, percentage, 0), "; impossible k:", pmf(2, percentage, 3))


if __name__ == "__main__":
    main()
code/python/probability_binomial_numpy.py
"""Vectorize binomial products without adding SciPy; comb stays exact."""
from fractions import Fraction as F
from math import comb
import numpy as np


def distribution(n, p):
    if type(n) is not int or not 0 <= n <= 30 or not np.isfinite(p) or not 0 <= p <= 1:
        raise ValueError("Require integer n in 0..30 and finite p in [0, 1]")
    k = np.arange(n + 1)
    coefficients = np.array([comb(n, int(j)) for j in k], dtype=float)
    # NumPy's exponent-zero result is 1, implementing the empty product.
    return coefficients * np.power(p, k) * np.power(1 - p, n - k)


def selected_mass(n, p, k):
    masses = distribution(n, p)
    if type(k) is not int or not -1 <= k <= 31:
        raise ValueError("Require integer k in -1..31")
    return 0.0 if k < 0 or k > n else float(masses[k])


def main():
    np.testing.assert_array_equal(distribution(2, 0.5), [0.25, 0.5, 0.25])
    assert selected_mass(3, 0.25, 1) == 27/64
    assert selected_mass(0, 0.4, 0) == 1
    assert selected_mass(0, 0.4, 1) == 0
    assert selected_mass(3, 0, 0) == selected_mass(3, 1, 3) == 1
    assert selected_mass(2, 0.5, -1) == selected_mass(2, 0.5, 3) == 0
    for n in range(31):
        for p_exact in (F(0), F(1, 10000), F(617, 5000), F(1, 2), F(9999, 10000), F(1)):
            masses = distribution(n, float(p_exact))
            exact = [comb(n, k) * p_exact**k * (1 - p_exact)**(n - k) for k in range(n + 1)]
            np.testing.assert_allclose(masses, [float(x) for x in exact], rtol=0, atol=1e-12)
            np.testing.assert_allclose(masses.sum(), 1, rtol=0, atol=1e-12)
            np.testing.assert_allclose(np.dot(np.arange(n + 1), masses), float(n * p_exact), rtol=0, atol=1e-12)
    for n, p, k in ((-1, .5, 0), (31, .5, 0), (2, np.nan, 0), (2, np.inf, 0), (2, 1.1, 0), (2, .5, 1.5)):
        try:
            selected_mass(n, p, k)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid binomial parameters accepted")
    print("Binomial(2, 1/2) ≈", distribution(2, .5))
    print("n = 30, p = 12.34%, k = 4 ≈", selected_mass(30, .1234, 4))


if __name__ == "__main__":
    main()

Explore

Hold n=2n=2 while moving pp from 00 to 11. Predict where mass collects at either end. Set p=1/2p=1/2 and increase nn: the support grows, but the selected event still asks for one count. Try n=0n=0 with both k=0k=0 and k=1k=1. Reset returns the stage’s starting model n=8n=8, p=35%p=35\%, k=3k=3.

For n=3n=3, p=25%p=25\%, select k=1k=1 and explain the factor 33 in the trace. Compare pp with 1−p1-p and reflect the count kk into n−kn-k: the success/failure roles swap, giving the same mass under the reflected parameter.

Common mistakes and counterexamples

Three possible counts for two trials are not equally likely. Counting labels would yield three masses 1/31/3 and discard sequence multiplicity. Also, pp is a probability, not the percentage number: 25%25\% means p=1/4p=1/4, not p=25p=25.

Equal trial marginals alone are insufficient. If the second trial always copies the first fair trial, the count is 00 or 22, each with mass 1/21/2, and never 11. Independent trials with differing success probabilities also need a different count model; one common pp cannot encode them.

Numerical considerations

Finite decimal percentages can be represented as exact rational inputs. Exact references avoid rounded factorial ratios by using integer combinations. The stage’s bounded n≤30n\le30 keeps its examples small, but floating-point masses remain approximations unless an exact form is supplied.

Boundary values deserve deliberate checks: n=0n=0, p=0p=0, p=1p=1, and impossible kk are valid cases with exact answers. Do not use a numerical epsilon to declare a small positive probability zero. General large-nn algorithms would require additional numerical care beyond this bounded lesson.

Engineering and ML connections

Binary indicators count errors, accepted requests, or successful tests. A binomial model is appropriate when independence and constant success probability are justified by the experiment. Reused inputs, common failures, or changing difficulty can violate those assumptions; a convenient formula does not repair the model.

Exercises

Enter exact fractions for the finite probabilities. Check boundaries using their experiment meaning as well as the formula.

calculation

Count sequences with equal success totals

Two independent Bernoulli(1/2) trials are summed. Give probabilities of 0, 1, and 2 successes.

calculation

Separate sequence mass from multiplicity

For X ~ Binomial(3, 1/4), find P(X = 1).

calculation

Handle zero trials and deterministic trials

Give P(X = 0) for Binomial(0, 2/5); P(X = 1) for that same model; P(X = 0) for Binomial(3, 0); P(X = 3) for Binomial(3, 1).

explanation

When can counts use the binomial formula?

Which experiment supports Binomial(n, p)?

Choose one answer

prediction

Do not clamp an impossible count

The stage has n = 2 and selected k = −1 or k = 3. What should it show?

Choose one answer

Summary and checkpoints

A Bernoulli variable records one binary event; a binomial variable counts independent trials with common pp. Check that you can explain both sequence probability and multiplicity, compute 27/6427/64 for the worked three-trial event, and handle zero trials, deterministic parameters, and impossible selected counts without changing the question.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available