Which part of a tiny probability can remain finite?

Intuition

A score offset can make every direct exponential enormous or tiny while leaving relative probabilities unchanged. Removing that common scale protects normalization. A very small final probability can still underflow, but its logarithm can be computed without constructing that tiny number.

Definition

For a nonempty score vector with at least one finite entry, let $m=\max_i z_i$. Then $\operatorname{LSE}(\mathbf z)=\log\sum_i e^{z_i}=m+\log\sum_i e^{z_i-m}$ and $\log p_i=z_i-\operatorname{LSE}(\mathbf z)$. Compute the latter equivalently as $(z_i-m)-\log\sum_j e^{z_j-m}$ to keep the correction after centering.

Mechanics

The table shows the actual input z, subtraction z−m, shifted exp, stable log probability and final probability for every class. The inspector displays the actual maximum, denominator, LSE and normalization residual. The naive denominator remains visible as a separate computation. Boundary presets present their actual exceptional vector; the model reports its policy without generating unrelated predictions.

Derivation

Factor out $e^m$ from the exponential sum. Taking log yields $m+\log\sum_i e^{z_i-m}$. Every finite shifted score is at most zero, at least one is zero, and the shifted denominator lies in [1,n]. Thus unnecessary exponential overflow disappears. For log-softmax, algebraically subtract the correction after centering; this avoids subtracting two huge rounded values. Computing $\log$ of a final rounded probability would discard information already present in the scores.

Worked examples

At (1000,999,998), m=1000, shifted scores (0,−1,−2), shifted exponentials $(1,e^{-1},e^{-2})$, denominator approximately 1.503215 and LSE approximately 1000.407606. At (1000,0,−1000), final binary64 probabilities are (1,0,0), but stable log probabilities remain (0,−1000,−2000). At two equal scores near $10^{308}$, the rounded LSE equals the common score; the centered log probabilities still equal $-\log2$.

Executable reference code

The standalone pedagogical implementation exposes the operations. Its NumPy companion uses array reductions and explicit domain/mask guards. Both include assertions for the hand-worked examples.

"""Expose every stable stage; retain centered log-softmax corrections."""
from math import exp,log,fsum,isfinite,inf
def pipeline(logits):
    if not 1<=len(logits)<=6 or any(not isfinite(z) and z!=-inf for z in logits):raise ValueError('Invalid logits')
    m=max(logits)
    if m==-inf:raise ValueError('All −infinity: no normalizable mass')
    shifted=[z-m for z in logits]
    exponentials=[exp(z) for z in shifted]
    denominator=fsum(exponentials);correction=log(denominator)
    log_probabilities=[z-correction for z in shifted]
    probabilities=[e/denominator for e in exponentials]
    return dict(max=m,shifted=shifted,exponentials=exponentials,denominator=denominator,logsumexp=m+correction,log_probabilities=log_probabilities,probabilities=probabilities)
if __name__=='__main__':
    d=pipeline([1000,0,-1000]);assert d['log_probabilities']==[0,-1000,-2000]
    assert d['probabilities']==[1,0,0]
    assert pipeline([1e308,1e308])['log_probabilities']==[-log(2),-log(2)]
    assert pipeline([0,-inf])['log_probabilities']==[0,-inf]
    print(pipeline([1000,0,-1000]))

"""Stable logsumexp and logsoftmax; logs do not take log of rounded probabilities."""
import numpy as np
def pipeline(logits):
    z=np.asarray(logits,dtype=float)
    if z.ndim!=1 or not 1<=z.size<=6 or np.any(np.isnan(z)) or np.any(z==np.inf):raise ValueError('Invalid logits')
    m=z.max()
    if m==-np.inf:raise ValueError('All excluded: normalization undefined')
    with np.errstate(over='ignore',under='ignore'):shifted=z-m;e=np.exp(shifted)
    denominator=e.sum();correction=np.log(denominator)
    return m+correction,shifted-correction,e/denominator
if __name__=='__main__':
    lse,lp,p=pipeline([1000,0,-1000]);assert np.array_equal(lp,[0,-1000,-2000]);assert np.array_equal(p,[1,0,0])
    assert np.array_equal(pipeline([1e308,1e308])[1],[-np.log(2),-np.log(2)])
    print(pipeline([1000,0,-1000]))

Explore

Select Overflow + tiny probability and identify every actual intermediate value. Next select Naive denominator underflows: the stable result still normalizes. Select One excluded class, All −∞, Reject +∞, Reject NaN and Empty vector; explain why they are different mathematical policies. Huge equal finite logits verifies that the log correction survives a huge common offset.

Common mistakes and counterexamples

Stabilization cannot guarantee that every tiny output probability is representable. Numerical probability zero from finite scores does not mean impossible support. A mathematical −∞ input explicitly excludes a class. All −∞ has a zero denominator and no unique distribution; inventing uniform probabilities would silently change the model. Positive infinity and NaN are rejected here rather than assigned an arbitrary limiting rule.

Numerical considerations

Empty vectors, more than six entries, NaN and +∞ are invalid. An isolated −∞ is accepted as excluded support; all −∞ is undefined. Finite max-shift subtraction can itself exceed binary64 range at extreme opposite scores; that −∞ log result is labeled numerical range loss, not mathematical impossibility. Stable output comes from the evaluated max-shift algorithm. Direct exponential normalization with 400-digit Decimal arithmetic provides an independent comparison for feasible scores; analytic equal/dominant identities provide references for enormous-score examples. Logs use a 2e−12 scaled absolute/relative comparison tolerance; probabilities use 2e−14 absolute error. Normalization budget is 32×epsilon×6. These are numerical allowances, not proofs of analytic identities.

Engineering and ML connections

Log-softmax lets a classifier or language model compute a finite NLL for a token whose displayed probability underflows. The next chapter selects the target’s stable log probability directly. Numerical stability changes the evaluation of the same mathematical model; it does not authorize smoothing impossible events.

Connect to softmax normalization.

Exercises

Predict intermediate quantities, calculate small cases and explain numerical or mask boundaries.

Summary and checkpoints

A score offset can make every direct exponential enormous or tiny while leaving relative probabilities unchanged. Removing that common scale protects normalization. A very small final probability can still underflow, but its logarithm can be computed without constructing that tiny number. Read the actual pipeline, preserve the log information and state which inputs contribute.