What does a prediction cost under the target distribution?

Intuition

Predicting an outcome poorly is costly when that outcome is common under the target. Cross entropy combines the target frequencies with the surprise assigned by a prediction. One observed label instead incurs just its own negative log probability.

Definition

For normalized discrete distributions P and Q on the same ordered support, $H(P,Q)=-\sum_i p_i\log q_i$. P is the target; Q is the prediction. A realized label y has negative log-likelihood $L_y=-\log q_y$. For a one-hot target $p_i=\mathbf1\{i=y\}$, cross entropy equals that realized NLL.

Mechanics

P starts fixed at (1/2,1/4,1/4). Only Q’s integer weights are editable and are normalized explicitly. The Target type selector deliberately replaces P by a one-hot observed class. The inspector keeps average cross entropy and the selected class’s NLL separate; the table and bars expose each actual contribution.

Derivation

Take the expectation under P of the prediction’s negative log score: $\mathbb E_P[-\log q_X]=\sum_i p_i(-\log q_i)$. In the one-hot case, every term except i=y has zero target weight, leaving $-\log q_y$. An iid labeled dataset uses a sum of realized NLLs, or a mean whose denominator counts its contributing observations. Empirical class frequencies recover cross entropy if the same Q is used for all observations.

Worked examples

For P=(1/2,1/4,1/4) and Q=(1/4,1/2,1/4), the bit contributions are (1,1/4,1/2), giving $H(P,Q)=7/4$ bits. If class 1 is the actual observed target, its NLL is 2 bits, which differs from that distribution average. For a target class with q=0, the NLL and one-hot cross entropy are infinite.

Executable reference code

The pedagogical version keeps exact source masses and branches on support. The NumPy version uses masks before logarithms and ratios. Both files include assertions for the hand-worked examples.

"""P weights the observations; Q supplies their predicted probabilities."""
from fractions import Fraction as F
from math import log, inf, fsum
def cross_entropy(p,q,unit='nats'):
    p=[F(str(x)) for x in p];q=[F(str(x)) for x in q]
    if not 1<=len(p)<=6 or len(p)!=len(q) or any(x<0 for x in p+q) or sum(p)!=1 or sum(q)!=1 or unit not in ('bits','nats'):raise ValueError('Same support; normalized distributions')
    scale=log(2) if unit=='bits' else 1
    terms=[0.0 if not a else inf if not b else -float(a)*(log(b.numerator)-log(b.denominator))/scale for a,b in zip(p,q)]
    return terms,fsum(terms)
if __name__=='__main__':
    assert cross_entropy([0,1,0],['1/2','1/4','1/4'],'bits')[1]==2
    assert cross_entropy([1,0],[0,1])[1]==inf
    assert cross_entropy([1,0],[1,0])[1]==0
    print(cross_entropy(['1/2','1/2'],['1/4','3/4']))

"""Vectorized target-weighted log loss with explicit zero-target masking."""
import numpy as np
def cross_entropy(p,q,unit='nats'):
    p=np.asarray(p,dtype=float);q=np.asarray(q,dtype=float)
    if p.ndim!=1 or not 1<=p.size<=6 or p.shape!=q.shape or not np.all(np.isfinite(p)) or not np.all(np.isfinite(q)) or np.any(p<0) or np.any(q<0) or abs(p.sum()-1)>1e-12 or abs(q.sum()-1)>1e-12 or unit not in ('bits','nats'):raise ValueError('Normalized finite distributions')
    terms=np.zeros_like(p);positive=p>0
    with np.errstate(divide='ignore'):terms[positive]=-p[positive]*np.log(q[positive])/(np.log(2) if unit=='bits' else 1)
    return terms,float(terms.sum())
if __name__=='__main__':
    assert cross_entropy([0,1,0],[.5,.25,.25],'bits')[1]==2
    assert cross_entropy([1,0],[0,1])[1]==np.inf
    assert cross_entropy([1,0],[1,0])[1]==0
    print(cross_entropy([.5,.5],[.25,.75]))

Explore

Select bits and compare the three contribution terms. Switch to one observed true class and move its predicted weight to zero. Change the target class explicitly and identify the one table row carrying the whole loss. Restore the distribution average and compare it with the realized NLL.

Common mistakes and counterexamples

H(P,Q) generally differs from H(Q,P). The target supplies weights, not logarithm arguments. A zero target mass contributes zero even when prediction is zero; never evaluate 0×log 0. A positive target mass at zero prediction contributes +∞. Epsilon smoothing is a changed model, not a free mathematical repair.

Numerical considerations

Both distributions are validated as exact rational masses with total exactly one. Predictions originate from visible nonnegative weights, whose all-zero vector is rejected. Logarithms are approximate; infinity arises from support mismatch rather than overflow. The finite sum is checked against Decimal100 at the declared 2e−12 scaled absolute/relative budget. The later logit pipeline computes NLL from stable log probabilities so probability underflow need not turn a finite NLL into infinity.

Engineering and ML connections

Classification trains predicted class probabilities against labels. Language models use a vocabulary prediction at each position and the observed next token as the label. Distribution-averaged cross entropy describes expected log loss; the training batch is a finite average over its actual observed targets. That scalar feeds gradient-based optimization.

Continue from the preceding information-theory mechanism.

Exercises

Compute actual terms and distinguish direction, support and observation from expectation.

Summary and checkpoints

Predicting an outcome poorly is costly when that outcome is common under the target. Cross entropy combines the target frequencies with the surprise assigned by a prediction. One observed label instead incurs just its own negative log probability. Always name the source and prediction before interpreting a loss.