Why can a positive probability disappear during multiplication?

Intuition

A long chain of likely events can have a tiny joint probability. The multiplication path can lose that positive value to underflow. Adding logs retains a useful finite score even when converting the score back to a probability underflows.

Definition

For independent observed events with probabilities $p_i\in[0,1]$, $L=\prod_i p_i$. For strictly positive factors, the natural log-likelihood is $\ell=\log L=\sum_i\log p_i$. Here a repeated block gives $L=(\prod_i p_i)^r$. Independence is an explicit modeling assumption; correlated events require conditional factors.

Mechanics

The three factor controls and repeated-block count own the state. The table shows exact rational factors and approximate log factors. The graph plots log-likelihood after each sampled block; the inspector shows the same final quantity alongside the actual direct multiplication result.

Derivation

The identity $\log(ab)=\log a+\log b$ follows because if $a=e^u$ and $b=e^v$, then $ab=e^{u+v}$. Repeating it yields $\ell$. Logs are increasing, so maximizing positive likelihood and maximizing its log choose the same parameters. Changing to negative log-likelihood turns that maximization into minimization.

Worked examples

At $p=(1/2,1/4,4/5)$, $L=1/10$ and $\ell=-\log 10\approx-2.302585$ nats. At three factors of $1/2$ repeated 400 times, $L=2^{-1200}>0$ and $\ell=-1200\log2\approx-831.776617$. Binary64 cannot represent that positive product, but the log remains finite.

Executable reference code

Run the standalone pedagogical version to see each operation; the NumPy companion shows the practical array form. Both include assertions so you can check the worked examples as you run them.

"""Exact independent-event product and approximate natural-log likelihood."""
from fractions import Fraction
from math import log, inf, prod, isfinite
def product(probabilities, repeat=1):
    if not 1 <= len(probabilities) <= 6 or type(repeat) is not int or not 1 <= repeat <= 1200:
        raise ValueError('Use 1–6 factors and 1–1200 repetitions')
    masses = [Fraction(str(p)) for p in probabilities]
    if any(p < 0 or p > 1 for p in masses): raise ValueError('Probabilities in [0,1]')
    exact = prod(masses) ** repeat
    log_likelihood = -inf if not exact else sum(log(p.numerator)-log(p.denominator) for p in masses) * repeat
    direct = 1.0
    for _ in range(repeat):
        for p in masses: direct *= float(p)
    return exact, direct, log_likelihood
if __name__ == '__main__':
    assert product([.5,.25,.8])[0] == Fraction(1,10)
    exact, direct, ll = product([.5,.5,.5],400)
    assert exact > 0 and direct == 0 and isfinite(ll)
    assert product([0,.5])[2] == -inf
    print(product([.5,.25,.8]))

"""NumPy binary64 multiplication versus log-space accumulation; zeros stay impossible."""
import numpy as np
def product(probabilities, repeat=1):
    p = np.asarray(probabilities, dtype=float)
    if p.ndim != 1 or not 1 <= p.size <= 6 or not np.all(np.isfinite(p)) or np.any((p<0)|(p>1)) or type(repeat) is not int or not 1<=repeat<=1200:
        raise ValueError('Finite probabilities in [0,1]; bounded repetition')
    with np.errstate(divide='ignore', under='ignore'):
        return float(np.prod(np.tile(p,repeat))), float(repeat*np.log(p).sum())
if __name__ == '__main__':
    assert np.isclose(product([.5,.25,.8])[0],.1)
    direct,ll=product([.5,.5,.5],400)
    assert direct==0 and np.isfinite(ll)
    assert product([0,.5])[1]==-np.inf
    print(product([.5,.25,.8]))

Explore

Predict what happens when one factor reaches zero. Then select Positive product underflows and distinguish the exact product from the displayed direct zero. Raise a probability and explain why the log-likelihood increases toward zero.

Common mistakes and counterexamples

Log probability is nonpositive; surprise is its negative. A finite log does not make an impossible event possible. Substituting epsilon for zero changes the model. Multiplying marginal probabilities without independence is not generally a joint probability.

Numerical considerations

Factors use exact decimal probabilities with at most two places; there are 1–6 supported factors and 1–1200 repetitions. Exact products remain BigInt fractions. Long exact products are displayed as an equivalent power of the exact block product. The approximate log is evaluated from the factors, independently checked against Decimal100 log of the exact product. Direct multiplication and exponentiating the log can underflow. The extended log of zero is −∞; negative, non-finite or out-of-range probability inputs are rejected.

Engineering and ML connections

The Statistics likelihood chapters treat observed data as fixed while candidate parameters vary. Log-likelihood makes long sequences additive. Classification and token loss will negate these log scores to give an optimization objective.

Continue from the preceding mathematical connection.

Exercises

Predict, compute and explain the actual quantities. Each question offers corrective feedback and a worked solution.

Summary and checkpoints

A long chain of likely events can have a tiny joint probability. The multiplication path can lose that positive value to underflow. Adding logs retains a useful finite score even when converting the score back to a probability underflows. Check the assumptions, unit and boundary before interpreting the computed result.