Why does comparison depend on direction?

Intuition

A prediction can place mass on an outcome absent from the source without being penalized there in the forward direction. Reversing the comparison changes which outcomes are weighted and which zero probabilities are fatal. Two tables with the same labels can therefore produce very different losses.

Definition

For normalized P and Q on a common finite support, $D_{\mathrm{KL}}(P\|Q)=\sum_i p_i\log(p_i/q_i)$. A p=0 term contributes zero without evaluating a ratio; p>0 and q=0 gives +∞. Natural logs give nats; base-2 logs give bits. KL is generally asymmetric and is not a distance metric.

Mechanics

P is fixed and Q is normalized from the editable weights. Both forward and reverse totals stay visible. Show Q || P deliberately changes the selected source, prediction, signed terms and entropy baseline. The selected identity residual checks finite floating-point evaluations without attempting infinity minus infinity.

Derivation

Expand the log ratio: $\sum_i p_i\log(p_i/q_i)=\sum_i p_i\log p_i-\sum_i p_i\log q_i=H(P,Q)-H(P)$. For non-negativity, assume Q is positive wherever P is positive. The inequality $\log t\le t-1$ gives $-D=\sum_{p_i>0}p_i\log(q_i/p_i)\le\sum_{p_i>0}(q_i-p_i)\le0$. Equality requires every positive-source ratio qᵢ/pᵢ=1 and no Q mass outside the source support, hence P=Q. If support is mismatched, D=+∞ and the inequality still holds. This derivation is independent of numerical examples.

Worked examples

For P=(1/2,1/4,1/4), Q=(1/4,1/2,1/4), signed bit terms are (1/2,−1/4,0), totaling 1/4 bit; cross entropy 7/4 minus entropy 3/2 agrees. For P=(1,0), Q=(1/2,1/2), forward KL is 1 bit and reverse is infinite. With Bernoulli P=(.1,.9), Q=(.2,.8), R=(.3,.7), $D(P\|R)\approx.116322$ exceeds $D(P\|Q)+D(Q\|R)\approx.062423$ nats, violating the triangle inequality.

Executable reference code

The pedagogical version keeps exact source masses and branches on support. The NumPy version uses masks before logarithms and ratios. Both files include assertions for the hand-worked examples.

"""Exact sources and direction-sensitive KL; preserve signed contribution terms."""
from fractions import Fraction as F
from math import log,inf,fsum
def kl(p,q,unit='nats'):
    p=[F(str(x)) for x in p];q=[F(str(x)) for x in q]
    if not 1<=len(p)<=6 or len(p)!=len(q) or any(x<0 for x in p+q) or sum(p)!=1 or sum(q)!=1 or unit not in ('bits','nats'):raise ValueError('Same support; normalized distributions')
    terms=[]
    for a,b in zip(p,q):
        if not a:terms.append(0.0)
        elif not b:terms.append(inf)
        else:
            ratio=a/b
            terms.append(float(a)*(log(ratio.numerator)-log(ratio.denominator))/(log(2) if unit=='bits' else 1))
    return terms,fsum(terms)
if __name__=='__main__':
    assert kl([1,0],['1/2','1/2'],'bits')[1]==1
    assert kl(['1/2','1/2'],[1,0],'bits')[1]==inf
    assert kl(['1/2','1/2'],['1/2','1/2'])[1]==0
    print(kl(['1/2','1/4','1/4'],['1/4','1/2','1/4']))

"""NumPy KL without ever evaluating zero divided by zero."""
import numpy as np
def kl(p,q,unit='nats'):
    p=np.asarray(p,dtype=float);q=np.asarray(q,dtype=float)
    if p.ndim!=1 or not 1<=p.size<=6 or p.shape!=q.shape or not np.all(np.isfinite(p)) or not np.all(np.isfinite(q)) or np.any(p<0) or np.any(q<0) or abs(p.sum()-1)>1e-12 or abs(q.sum()-1)>1e-12 or unit not in ('bits','nats'):raise ValueError('Normalized finite distributions')
    terms=np.zeros_like(p);positive=p>0
    with np.errstate(divide='ignore'):terms[positive]=p[positive]*(np.log(p[positive])-np.log(q[positive]))/(np.log(2) if unit=='bits' else 1)
    return terms,float(terms.sum())
if __name__=='__main__':
    assert kl([1,0],[.5,.5],'bits')[1]==1
    assert kl([.5,.5],[1,0])[1]==np.inf
    assert kl([.5,.5],[.5,.5])[1]==0
    print(kl([.5,.25,.25],[.25,.5,.25]))

Explore

Compare the canonical forward and reverse values, then choose Unequal forward / reverse and change individual Q weights. Use Zero predicted true-class mass and inspect which direction becomes infinite. Finally choose Q equals P and verify zero divergence without zeroing the entropy.

Common mistakes and counterexamples

Individual KL terms can be negative; the complete normalized sum is nonnegative. KL is not an entropy and is not a metric: it lacks symmetry and the triangle inequality. Zero/zero must be skipped before division. Comparing reversed arguments requires reversing the entropy baseline as well.

Numerical considerations

Exact rational masses and support decisions are preserved. Positive rational ratios are formed exactly before their approximate logarithm is taken. For an independent numerical comparison of finite cases, use 100-digit Decimal log differences and separately total CE−H. Finite results and identity residuals use a 2e−12 scaled absolute/relative budget. Tiny signed rounding residuals are not a mathematical violation of non-negativity; no sampled plot is a proof.

Engineering and ML connections

Expected prediction log loss is minimized by predicting P itself: cross entropy equals H(P) plus nonnegative KL. This connects probabilistic training loss with information mismatch. Direction remains essential in inference and model approximation; a symmetric-looking picture cannot make it a distance.

Continue from the preceding information-theory mechanism.

Exercises

Compute actual terms and distinguish direction, support and observation from expectation.

Summary and checkpoints

A prediction can place mass on an outcome absent from the source without being penalized there in the forward direction. Reversing the comparison changes which outcomes are weighted and which zero probabilities are fatal. Two tables with the same labels can therefore produce very different losses. Always name the source and prediction before interpreting a loss.