Which curve is which?

Try it before reading on. Switch the curves on one at a time and drag the input $x$ along the axis, then press who wins? and watch how quickly $e^{x}$ leaves the plane on the right. Three of these curves, $x$, $x^{2}$ and $e^{x}$, all climb; they do not climb in the same way. Switch on $\ln x$ and drag $x$ to the left of zero: its row turns into a message, not a number. By the end of the chapter you will be able to say which curve is which from its shape, to read what two parameters do to any of them, and to say exactly where each stops being defined and where each stops being computable on a real machine.

Intuition: shape first

Most of the functions an engineer or a machine-learning practitioner meets are built from six simple shapes. Learn the shapes first; the formulas then explain them.

• The line, $y=x$. Steady change: each step right adds the same amount. Its graph is straight.
• The parabola, $y=x^{2}$. Change that speeds up away from the bottom. It is symmetric: $x$ and $-x$ give the same value, so squaring forgets signs.
• The exponential, $y=e^{x}$. Change in proportion to the amount already there. It is always positive, it never reaches the axis, and on the right it grows faster than any line or parabola, although not immediately.
• The logarithm, $y=\ln x$. The exponential read backwards: it asks what power of $e$ gives $x$? It exists only for $x>0$, falls without bound as $x$ approaches $0$, and climbs more and more slowly to the right.
• The sine and the cosine, $y=\sin x$ and $y=\cos x$. A wave that repeats every $2\pi$ and never leaves the band from $-1$ to $1$. The cosine is the sine started a quarter of a period earlier.

Here $e\approx 2.718282$ is the number for which the exponential has the pleasant property that its steepness at any point equals its height there. The book writes $\approx$ because $e$ is irrational and the digits are a rounding; the stage above uses the same rule, and it writes $=$ only for values that are exactly known.

Definition

All six share one pattern. Take a basic function $g$ from the list and two numbers $a$ and $h$:

The number $a$ is the vertical scale: it stretches the graph away from the $x$-axis, and a negative $a$ also flips it upside down. The number $h$ is the horizontal shift: the graph of $g(x-h)$ is the graph of $g$ moved right by $h$, which is the opposite of what the minus sign tempts you to expect (the value $g$ had at $0$ is now found at $x=h$).

Every function here has a domain, the inputs for which it exists, and a range, the outputs it can produce. These come from $g$ and the shift, and they are the first thing to know:

• $x$ and $x^{2}$ and $e^{x}$ and $\sin x$ and $\cos x$ accept every real number; $\ln x$ accepts only $x>0$, so $a\ln(x-h)$ exists only for $x>h$, with a vertical asymptote at $x=h$.
• The range of $a\sin(x-h)$ is the interval from $-|a|$ to $|a|$. The range of $a\,e^{x-h}$ is the positive numbers when $a>0$ and the negative numbers when $a<0$. The range of $a\ln(x-h)$ is every real number.
• If $a=0$ the result is the constant $0$ (where it exists), whichever $g$ you chose. The stage flags that as a special case, because the shape disappears.

An exponential has a small surprise hidden in this pattern: shifting it and scaling it are the same operation, because $a\,e^{x-h}=(a\,e^{-h})\,e^{x}$. Slide $h$ and watch the curve; then set $a$ close to $e^{-h}$ (about $0.37$ for $h=1$) and the two curves become indistinguishable. Exactly equal needs $a=e^{-h}$, an irrational number that the two-decimal field cannot hold. No other family does this.

Mechanics: shifts, scales and inverses

Shifts and scales

Use the stage at the top (or the full copy in the Explore section) to see the two parameters act. Set only $\ln$ on, then raise $h$: the curve and its vertical asymptote move right together, and the thick bar on the $x$-axis, where $\ln$ has no value, grows with them. Set only the sine on and raise $a$ to $3$: the range becomes $[-3,3]$ and the zeros do not move, because a vertical stretch never moves the points where a function is $0$. Raise $h$ instead and the zeros slide.

Inverse pairs and their honest domains

Two functions are inverses when each undoes the other. The exponential and the logarithm are one such pair, and squaring and the square root are another. On a graph, inverses are mirror images across the line $y=x$, because undoing a function swaps the roles of input and output, and swapping coordinates is exactly that mirror.

The honest statement of the pair needs the domains:

The first holds everywhere because $e^{x}$ is defined and positive for every $x$. The second needs $x>0$ because $\ln x$ does not exist otherwise. The square-and-root pair has the same trap in a different place: $\sqrt{x^{2}}=|x|$, which equals $x$ only when $x\ge0$, and $\left(\sqrt{x}\right)^{2}=x$ only for $x\ge0$, since $\sqrt{x}$ does not exist for negative $x$.

Try the presets in turn. $\ln(e^{1.5})$ comes back, and so does $\ln(e^{-3})$: negative inputs are fine for the exponential. $e^{\ln(-2)}$ cannot even start, because the first step has no value. $\sqrt{(-2)^{2}}$ comes back as $2$, not $-2$: the square forgot the sign, and the stage shows where the lost point lands on the other branch of the parabola. And the two overflow presets show that a round trip that is mathematically perfect can still fail on a computer; the row says so, labelled “as a double”.

The unit circle

Trigonometry begins with one circle. The unit circle is the circle of radius $1$ centred at the origin. Start at $(1,0)$ and walk counter-clockwise a distance $\theta$ along the circle; the point you reach is, by definition,

The distance walked, $\theta$, is measured in radians: one full turn is $2\pi$ because the circumference of a circle of radius $1$ is $2\pi$. So $\pi$ is a half-turn, $\pi/2$ a quarter-turn, and $180^{\circ}=\pi$ radians. Walking clockwise is a negative angle, matching the convention used for vector directions in the vectors chapter. Walking a whole turn more brings you back to the same point, which is why both functions repeat: $\sin(\theta+2\pi)=\sin\theta$.

Because the point lies on the circle, its coordinates satisfy $\cos^{2}\theta+\sin^{2}\theta=1$: it is Pythagoras applied to the right triangle with the radius as hypotenuse.

Some angles have exact coordinates built from $\sqrt2$, $\sqrt3$ and small fractions. The stage in its exact mode snaps to multiples of $\pi/12$ and shows them with $=$. The table lists the first half-turn, straight from the model:

The other quadrants follow from the first by reflection. Take the reference angle, the acute angle between the radius and the $x$-axis, read its unsigned values from the table, and put the signs back from the quadrant: in quadrant II cosine is negative and sine positive, in quadrant III both are negative, in quadrant IV cosine is positive and sine negative.

Two inverse functions go with the circle, and both have limited ranges. $\arcsin$ returns only angles from $-\pi/2$ to $\pi/2$, so $\arcsin(\sin\theta)=\theta$ only for those $\theta$; for $\theta=2$ it returns $\pi-2$, the other angle with the same sine. The two-argument function $\operatorname{atan2}(y,x)$, met in the vectors chapter, returns the angle in $(-\pi,\pi]$ from both coordinates, so it recovers $\theta$ up to whole turns. The verdict line under the stage says, for each angle, whether $\arcsin$ and $\operatorname{atan2}$ give the angle back.

Changing the base

The exponential $e^{x}$ is one member of a family. For any base $b>0$ with $b\ne1$ we have $b^{x}=e^{x\ln b}$, so every exponential is the natural one with the input rescaled by the constant $\ln b$. The inverse of $b^{x}$ is the logarithm to the base $b$, written $\log_{b}x$: the power of $b$ that gives $x$. It exists for $x>0$ only.

Base $1$ is excluded because $1^{x}=1$ for every $x$: it is a constant, not an exponential, and its “logarithm” would have to answer what power of $1$ gives $7$? — nothing does. For a base between $0$ and $1$ the curve falls instead of rising (halving rather than doubling).

Any logarithm is a natural logarithm divided by a constant, which is how a computer evaluates all of them:

The same constant gives the doubling time. For $b>1$ the quantity $b^{x}$ doubles every time $x$ grows by $\ln 2/\ln b$, and for $0<b<1$ it halves every $\ln 2/|\ln b|$. And it converts units of information: $\log_{2}x=\ln x/\ln 2$ counts bits and $\ln x$ counts nats.

Finally, these identities hold for positive numbers and are what make logarithms useful (the last is the special case $b=2$ of the third, which is how bits are computed):

Exact mathematics says both sides of each identity are equal. The computer rounds every step, so the two sides can differ in the last digits; the stage shows that difference and a tolerance of a few units in the last place. The checker also shows when an identity cannot be checked at all: if $a$ is not positive, $\ln a$ does not exist and the card says so, rather than inventing a number.

Worked examples

Four computations, each shown step by step with every number taken from the same model as the stages. Step forward and back with the buttons; going back selects an earlier step, and nothing is undone.

Executable reference code

Run the Python examples below to follow the calculations step by step. The comments record what each line returns.

First the exponential and the logarithm: the round trip, the places where each fails, a square and its root, change of base, and bits and nats. Notice the line math.log(1000, 10): it returns 2.9999999999999996, because it divides two rounded logarithms, while math.log10(1000.0) is built for the job and returns 3.0.

import math

# ln undoes e^x for EVERY real x: ln(e^x) = x. The computer rounds twice, so the answer can differ in the last digits.
x = 1.5
back = math.log(math.exp(x))  # 1.5 (here it happens to come back exactly)

# e^x is larger than the largest double (about 1.8e308) once x > 709.78. Python raises an error; NumPy would return inf.
try:
    math.exp(710.0)
except OverflowError:
    overflow = "OverflowError"

# ln x does not exist for x <= 0: no power of e is zero or negative.
try:
    math.log(0.0)
except ValueError:
    no_log_of_zero = "ValueError"

# e^(ln x) = x only for x > 0. At x = 10 the round trip is a hair off, because ln 10 is stored rounded.
exp_of_ln = math.exp(math.log(10.0))  # 10.000000000000002: compare with math.isclose, never ==
exp_of_ln_close = math.isclose(exp_of_ln, 10.0, rel_tol=1e-12)  # True

# Square then root: sqrt(x**2) is |x|, not x. The sign is lost on the way up.
sqrt_of_square = math.sqrt((-3.0) ** 2)  # 3.0

# Change of base: log_b a = ln a / ln b. In floating point the quotient is not always the integer you expect.
log2_of_8 = math.log2(8.0)  # 3.0, exactly (math.log2 is built for powers of two)
log_1000_base_10 = math.log(1000, 10)  # 2.9999999999999996 (a quotient of two rounded logarithms)
log10_of_1000 = math.log10(1000.0)  # 3.0 (the dedicated function)

# Bits and nats: log2(a) = ln(a) / ln(2).
bits = math.log(1024.0) / math.log(2.0)  # 10.0
nats = math.log(1024.0)  # 6.931471805599453

# A doubling time: how far must x go for 3**x to double?  log_3(2) = ln 2 / ln 3.
doubling_time = math.log(2.0) / math.log(3.0)  # 0.6309297535714574

print(back, overflow, no_log_of_zero, exp_of_ln, exp_of_ln_close, sqrt_of_square, log2_of_8, log_1000_base_10, log10_of_1000, bits, nats, doubling_time)

Then trigonometry. All Python trigonometric functions take radians. The file shows that $\sin\pi$ is not zero (it is about $1.2\times10^{-16}$), why you compare with a tolerance, and the two behaviours from the unit circle: $\arcsin(\sin 2)$ is not $2$, and atan2 returns the angle modulo a turn.

import math

# Python's trigonometric functions take RADIANS. A full turn is 2*pi.
theta = math.pi / 6
cos_t, sin_t = math.cos(theta), math.sin(theta)  # 0.8660254037844387 and 0.49999999999999994: sqrt(3)/2 and 1/2 up to rounding

# cos^2 + sin^2 = 1 holds exactly in mathematics; the doubles agree to about 1e-16, so compare with isclose.
pythagorean_gap = cos_t**2 + sin_t**2 - 1.0
pythagorean_ok = math.isclose(cos_t**2 + sin_t**2, 1.0, rel_tol=1e-15)

# pi is stored rounded, so sin(pi) is not 0 and cos(pi/2) is not 0. Never test a trigonometric value with ==.
sin_pi = math.sin(math.pi)  # 1.2246467991473532e-16
cos_half_pi = math.cos(math.pi / 2)  # 6.123233995736766e-17
sin_pi_is_zero = math.isclose(sin_pi, 0.0, abs_tol=1e-12)  # True: ask for a tolerance

# Periodicity: adding a full turn gives the same point (to rounding).
periodic = math.isclose(math.sin(theta + 2 * math.pi), math.sin(theta), rel_tol=0, abs_tol=1e-15)

# arcsin(sin t) is NOT t unless t is in [-pi/2, pi/2]: arcsin only returns angles in that interval.
arcsin_of_sin_2 = math.asin(math.sin(2.0))  # 1.1415926535897933, which is pi - 2 up to rounding

# atan2(sin t, cos t) IS t modulo 2*pi, in (-pi, pi]: it sees the signs of both components.
atan2_of_4 = math.atan2(math.sin(4.0), math.cos(4.0))  # -2.2831853071795867, which is 4 - 2*pi up to rounding

# Degrees and radians: 180 degrees is pi radians.
thirty_degrees = math.radians(30.0)  # 0.5235987755982988
back_to_degrees = math.degrees(math.pi / 6)  # 29.999999999999996: close to 30, not equal

print(cos_t, sin_t, pythagorean_ok, sin_pi, cos_half_pi, sin_pi_is_zero, periodic, arcsin_of_sin_2, atan2_of_4, thirty_degrees, back_to_degrees)

Finally the practical form of the exponential in machine learning: the softmax, which overflows if written naively, and the standard shift that fixes it.

import numpy as np

# Scores ("logits") from a model. Softmax turns them into probabilities: exp(s_i) / sum_j exp(s_j).
logits = np.array([1000.0, 1001.0, 1002.0])

# e^1000 is far above the largest double (about 1.8e308), so NumPy returns inf, and inf / inf is nan.
with np.errstate(over="ignore", invalid="ignore"):
    naive = np.exp(logits) / np.exp(logits).sum()  # [nan nan nan]

# The probabilities do not change if every score is shifted by the same amount: e^(s - m) / sum e^(s_j - m) = e^s / sum e^(s_j).
# Shift by the largest score m, so the largest exponent is e^0 = 1 and nothing can overflow.
shifted = np.exp(logits - logits.max())
stable = shifted / shifted.sum()  # [0.09003057 0.24472847 0.66524096]

# The same idea for log(sum(exp(s))): m + log(sum(exp(s - m))).
log_sum_exp = logits.max() + np.log(shifted.sum())  # 1002.4076059644444

print(naive, stable, log_sum_exp)

Explore: change the parameters

This is the whole comparison on one plane, with the facts about each visible curve in cards underneath: domain, range, zeros, asymptote, behaviour and symmetry. These questions have definite answers; check them against the stage.

• Switch on $x$, $x^{2}$ and $e^{x}$ together with $a=1$ and $h=0$. Where do the line and the parabola cross? Does $e^{x}$ ever fall below $x^{2}$? (For positive $x$ it never does: the smallest ratio $e^{x}/x^{2}$ is $e^{2}/4\approx1.85$, at $x=2$. For negative $x$ it does, until they cross near $x\approx-0.70$.) The exercise below asks the same question for $2^{x}$, where the answer is different.
• Turn $\ln$ on and raise $h$ to $3$. Where is its zero, and where is its asymptote? Why is the zero not at the asymptote?
• Set $a=-1$ on the parabola. What happens to its range, and to the facts card?
• Turn on the sine and cosine with $a=1$. Which value of $h$ for the sine brings it closest to the cosine? (Think of a quarter of a period, and of the direction a positive $h$ moves a graph. The exact answer is $-\pi/2\approx-1.571$; the arrows and the buttons step by $0.5$, so they reach $-1.5$; typing a value allows two decimals.)
• Make $a=0$. What do all the families have in common?

Common mistakes

Numerical considerations

The exponential has a narrow window in which a double can hold it. The largest double is about $1.8\times10^{308}$, so $e^{x}$ overflows just above $x=709.78$; the smallest positive double is about $5\times10^{-324}$, so $e^{x}$ underflows to exactly $0$ just below $x=-745.13$. The table is computed by the same model as the stages above.

When the stored value is $\infty$ or $0$, the stage writes $=$ because it is stating a fact about the double (“the stored value is $\infty$”), and it adds a note giving the truth: the real value is finite and large, or positive and tiny. Python and NumPy even disagree on what to do. math.exp(710.0) raises OverflowError; np.exp(710.0) quietly returns inf, and inf / inf is nan, which then poisons every number it touches.

Three more places where the machine and the mathematics part ways. First, $e^{\ln x}$ is $x$ only up to rounding: math.exp(math.log(10.0)) is 10.000000000000002. Second, $\pi$ is stored rounded, so $\sin\pi$ is about $1.2\times10^{-16}$, not $0$, and $\cos(\pi/2)$ is about $6\times10^{-17}$. Third, math.log(1000, 10) is 2.9999999999999996 but math.log10(1000.0) is 3.0: when a dedicated function for your base exists, use it.

Connections to engineering and machine learning

• Softmax and log-sum-exp. A classifier turns scores $s_i$ into probabilities $e^{s_i}/\sum_j e^{s_j}$. With scores near $1000$ the exponentials overflow. Subtracting the largest score $m$ multiplies the top and the bottom by the same number $e^{-m}$, so the probabilities do not change, and every exponent becomes at most $0$. The same shift gives $\ln\sum_j e^{s_j}=m+\ln\sum_j e^{s_j-m}$, the standard stable form used in the loss functions of almost every framework.
• Logarithms turn products into sums. The likelihood of $1000$ independent events each with probability $0.01$ is $10^{-2000}$, which underflows to $0$ in a double. The log-likelihood is $1000\ln 0.01\approx-4605.17$, an ordinary number. This is why models are trained on the log-likelihood.
• Bits and nats. Cross-entropy and information are measured in nats when computed with $\ln$ (the default in most libraries) and in bits with $\log_2$; one nat is $1/\ln2\approx1.4427$ bits. Telling $1024$ equally likely cases apart takes $\log_21024=10$ bits.
• Exponential growth and decay. Compound growth, radioactive decay, learning-rate schedules and the attenuation of a signal are all $a\,b^{x}$; the doubling or halving time $\ln2/|\ln b|$ is the one number that describes them.
• Sine and cosine. Every periodic signal is built from them, and a rotation of the plane by $\theta$ is the matrix whose entries are $\cos\theta$ and $\sin\theta$. Positional encodings in transformers use sines and cosines of many frequencies; all of them take radians.
• The logarithm as a compressor. On a log axis, multiplying a quantity by a fixed factor moves it a fixed distance. This is why spectra, learning curves and file sizes are plotted that way.

Exercises

Answer each question and press Check. A wrong answer tells you which mistake it looks like. After one attempt you may open a worked solution.

Summary and checkpoints

• Six shapes carry most of the work: the line, the parabola, the exponential, the logarithm, the sine and the cosine. Each is $a\,g(x-h)$ for a basic $g$: $a$ stretches vertically, $h$ shifts right.
• Domains matter. $\ln$ exists only for positive inputs; $\sqrt{}$ only for non-negative ones. $\ln(e^{x})=x$ for every $x$, but $e^{\ln x}=x$ only for $x>0$, and $\sqrt{x^{2}}=|x|$.
• Inverse functions are mirror images across $y=x$.
• On the unit circle, $(\cos\theta,\sin\theta)$ is the point at distance $\theta$ (radians) from $(1,0)$, and $\cos^{2}\theta+\sin^{2}\theta=1$. $\arcsin$ returns only angles in $[-\pi/2,\pi/2]$; $\operatorname{atan2}$ returns angles in $(-\pi,\pi]$.
• $\log_{b}x=\ln x/\ln b$; the doubling time is $\ln2/\ln b$; bits use base $2$, nats use base $e$.
• In code: radians everywhere, tolerances instead of ==, shift exponents before using them, and expect exp to overflow near $709.78$.

Checkpoints. You are ready to move on when you can, without looking:

1. Sketch $\ln(x-2)$ and name its domain, its asymptote and its zero.
2. Say why $\sqrt{x^{2}}$ is not always $x$, and for which $x$ it is.
3. Give $\cos$ and $\sin$ of $7\pi/6$, with signs, and say which quadrant that is.
4. Explain why $\arcsin(\sin 2)\ne2$, and what it equals.
5. Compute $\log_{2}1024$ and explain how it differs from $\ln1024$.
6. Say what happens to $e^{710}$ in Python and in NumPy, and how to avoid the overflow in a softmax.