How do unconstrained scores become probabilities?

Intuition

Scores can be negative or larger than one because they are not probabilities. Exponentiating makes relative scores positive; dividing by their total makes the outputs sum to one. A common offset changes the exponential scale but cancels out of the mathematical probabilities.

Definition

For a nonempty finite logit vector $\mathbf z\in\mathbb R^n$, $p_i=\operatorname{softmax}(\mathbf z)_i=e^{z_i}/\sum_j e^{z_j}$. Every mathematical output is positive and their sum is one. An optional explicit −∞ score denotes an excluded class with zero mass; at least one score must remain finite.

Mechanics

Each score control changes one entry in the authoritative vector. The table exposes actual naive exponentials and probabilities alongside stable output. The inspector exposes the naive denominator, shifted denominator, log-sum-exp and selected class probability. Common finite shift changes the effective input vector, with a displayed probability difference relative to the original scores.

Derivation

Sum the output probabilities to obtain $\sum_i e^{z_i}/\sum_j e^{z_j}=1$. For a common finite shift c, $e^{z_i+c}=e^ce^{z_i}$, so the common factor cancels: $\operatorname{softmax}(\mathbf z+c\mathbf1)=\operatorname{softmax}(\mathbf z)$. The same cancellation permits subtraction of the largest score before exponentiation.

Worked examples

For logits $(\log2,0)$, exponentials are (2,1), denominator 3 and probabilities (2/3,1/3). For equal logits (0,0,0), outputs are (1/3,1/3,1/3). The canonical scores (2,1,0) produce approximately (0.665241,0.244728,0.090031). Adding 1000 preserves the mathematical ratios even though direct exponentials overflow.

Executable reference code

The standalone pedagogical implementation exposes the operations. Its NumPy companion uses array reductions and explicit domain/mask guards. Both include assertions for the hand-worked examples.

"""Scores become positive exponentials and then normalized probabilities."""
from math import exp,fsum,isfinite,inf
def softmax(logits):
    if not 1<=len(logits)<=6 or any(not isfinite(z) and z!=-inf for z in logits):raise ValueError('Finite scores or explicit −infinity')
    m=max(logits)
    if m==-inf:raise ValueError('All excluded: normalization undefined')
    exponentials=[exp(z-m) for z in logits]
    denominator=fsum(exponentials)
    return exponentials,denominator,[e/denominator for e in exponentials]
if __name__=='__main__':
    assert softmax([0,0])[2]==[.5,.5]
    assert softmax([1,2,3])[2]==softmax([101,102,103])[2]
    assert softmax([1000,0,-1000])[2]==[1,0,0]
    print(softmax([2,1,0]))

"""NumPy score normalization with max-shifting and declared input policies."""
import numpy as np
def softmax(logits):
    z=np.asarray(logits,dtype=float)
    if z.ndim!=1 or not 1<=z.size<=6 or np.any(np.isnan(z)) or np.any(z==np.inf):raise ValueError('Invalid logits')
    if np.all(z==-np.inf):raise ValueError('All excluded: normalization undefined')
    with np.errstate(under='ignore',over='ignore'):e=np.exp(z-z.max())
    return e,float(e.sum()),e/e.sum()
if __name__=='__main__':
    assert np.array_equal(softmax([0,0])[2],[.5,.5])
    assert np.allclose(softmax([1,2,3])[2],softmax([1001,1002,1003])[2])
    print(softmax([2,1,0]))

Explore

Predict which class gets the largest mass. Change one score through the keyboard and numeric field, then apply a common shift. Select Overflow + tiny probability and compare the naive denominator failure with the stable prediction. Use Equal logits to confirm the uniform output.

Common mistakes and counterexamples

Logits need not sum to one, and negative logits are not negative probabilities. Softmax is invariant to a common additive shift, not to arbitrary changes of relative score or multiplication by a common factor. The selected class is a user-chosen index; it is not secretly replaced by the argmax.

Numerical considerations

The bounded explorer has 1–6 model classes and three canonical editable classes. Finite controls accept two-place scores and shifts in [−1000,1000]. Real exponentials are finite for finite scores, but binary64 exp(1000) overflows. Stable shifted exponentials are at most one. Normalization is checked at 32×epsilon×6, about 4.27e−14; independent Decimal400 reference probabilities have a separate 2e−14 absolute comparison tolerance. Adding a huge common shift in binary64 can round away the original score differences, so invariance is a mathematical identity with a numerical input-rounding caveat.

Engineering and ML connections

Classification and vocabulary prediction use relative scores to define categorical probabilities. A logit is an input to this transformation, not an event probability. Cross entropy scores the resulting prediction against a target, and the next chapter retains log probabilities without taking a logarithm of underflowed outputs.

Connect to the accepted prerequisite mechanism.

Exercises

Predict intermediate quantities, calculate small cases and explain numerical or mask boundaries.

Summary and checkpoints

Scores can be negative or larger than one because they are not probabilities. Exponentiating makes relative scores positive; dividing by their total makes the outputs sum to one. A common offset changes the exponential scale but cancels out of the mathematical probabilities. Read the actual pipeline, preserve the log information and state which inputs contribute.