Why is a smaller step not always a better derivative?

For $\sin x$ at $x=1$, the forward difference with $h=10^{-4}$ is off by about $4\times10^{-5}$ and the centered difference by about $9\times10^{-10}$. Drag toward smaller $h$: both errors fall at first, reach a minimum, then rise. At $h=10^{-14}$ the forward estimate is wrong in the third digit. Exact algebra says smaller $h$ is better; floating-point arithmetic disagrees.

Intuition: two errors pulling in opposite directions

A difference quotient replaces a limit by a finite step, which costs truncation error: the secant is not the tangent. Shrinking $h$ reduces it. But $f(x+h)$ and $f(x)$ are each rounded to about 16 significant digits; when $h$ is tiny they agree in most of those digits, their difference keeps only a few correct ones, and dividing by $h$ magnifies the loss. That rounding error grows as $h$ shrinks. The total is smallest somewhere in between.

Definition

With step $h>0$:

These are the forward, backward and centered differences. Each approximates $f'(x)$; their errors are measured here against an independent reference: the analytic derivative ($\cos x$ for $\sin x$, $e^x$ for $e^x$, and the exact rational $x^2/2-1$ for the cubic).

Mechanics: what the explorer computes

All quotients are computed in IEEE binary64, exactly as naive code would, dividing by the nominal $h$. For each scheme and each $k=1,\dots,16$ the plot shows $\log_{10}$ of the observed error. The dashed budget curves add the leading truncation term and a rounding term built from machine epsilon $\varepsilon=2^{-52}$ and the sizes of the sampled values. The budgets are explanations and upper-envelope estimates for these smooth functions, not guarantees for arbitrary $f$.

Derivation: the two terms and the best step

Taylor’s theorem gives $f(x+h)=f(x)+hf'(x)+\tfrac{h^2}2f''(\xi)$, so

a truncation error of order $h$.

For the centered difference the $h^2$ terms cancel and the error is $\tfrac{h^2}6f'''(\eta)$ — order $h^2$. Each computed value carries a relative rounding error of about $\varepsilon$, so the computed numerator is off by roughly $\varepsilon(|f(x+h)|+|f(x)|)$, divided by $h$. Minimising

gives $h^*\approx2\sqrt{\varepsilon|f|/|f''|}$, about $10^{-8}$ for well-scaled $f$; for the centered difference $h^*\approx\sqrt[3]{3\varepsilon|f|/|f'''|}$, about $10^{-5}$.

Worked example

$f=\sin$, $x=1$, $f'(1)=\cos1\approx0.5403023$:

• $h=10^{-1}$: forward error $\approx4.3\times10^{-2}$, centered $\approx9\times10^{-4}$ — truncation dominates;
• $h=10^{-4}$: forward $\approx4\times10^{-5}$, centered $\approx9\times10^{-10}$;
• $h=10^{-8}$: forward $\approx3\times10^{-9}$, the best region for forward differences;
• $h=10^{-14}$: forward $\approx4\times10^{-3}$ — rounding has taken over.

When numbers mislead

At $h=10^{-16}$, $x+h$ may round to $x$ itself, making $D_+=0$ — a confident, completely wrong answer. Even the step is inexact: $(x+h)-x$ is usually not exactly $h$ in binary64, a representation error the budget includes. For a function with a corner or a jump, no step size gives a meaningful derivative, because none exists (III.3).

Executable reference code

The reference compares the three schemes with an exact derivative of a polynomial evaluated in floating point and in Fraction, isolating rounding error. The NumPy version sweeps $h=10^{-1},\dots,10^{-16}$, finds the empirical best step and compares it with $\sqrt\varepsilon$.

"""Finite differences of a polynomial: exact rational quotients versus binary64 ones isolate rounding error (III.14)."""
from fractions import Fraction as F

p = lambda x: x**3 / 6 - x                  # works for Fraction and float
dp = lambda x: x**2 / 2 - 1                 # exact derivative

x = F(3, 2)
for k in range(1, 17):
    h = F(1, 10**k)
    exact_forward = (p(x + h) - p(x)) / h          # exact rational quotient
    # In exact arithmetic the forward error is exactly h*x/2 + h**2/6 (truncation only).
    assert exact_forward - dp(x) == h * x / 2 + h * h / 6

xf = 1.5
errs = [abs((p(xf + 10.0**-k) - p(xf)) / 10.0**-k - float(dp(x))) for k in range(1, 17)]
best = min(range(16), key=lambda i: errs[i])
assert 5 <= best + 1 <= 10                # the best step is near sqrt(eps) ~ 1e-8, not the smallest one
assert errs[15] > errs[best] * 100        # h = 1e-16 is far worse than the best step

"""Sweep h = 10**-1 ... 10**-16 for sin at 1: forward error is smallest near sqrt(eps), centered near eps**(1/3) (III.14)."""
import numpy as np

x = 1.0
h = 10.0 ** -np.arange(1, 17)
forward = np.abs((np.sin(x + h) - np.sin(x)) / h - np.cos(x))
centered = np.abs((np.sin(x + h) - np.sin(x - h)) / (2 * h) - np.cos(x))
eps = np.finfo(float).eps
assert abs(np.log10(h[np.argmin(forward)]) - np.log10(np.sqrt(eps))) <= 1.5
assert abs(np.log10(h[np.argmin(centered)]) - np.log10(np.cbrt(eps))) <= 1.5
assert centered.min() < forward.min()          # higher order wins at its best step
assert forward[-3] > 100 * forward.min()       # tiny steps are much worse

Explore

Compare eˣ at 2 with sin x: the larger $|f|$, the larger the rounding term and the earlier the error curve turns up. On the polynomial, the centered difference’s truncation error is tiny; its curve is shaped almost entirely by rounding. Notice where forward and backward errors differ in sign but not in size.

Common mistakes

Choosing $h$ as small as possible. Using a fixed absolute $h$ for inputs of very different magnitude (a relative step such as $h=\sqrt\varepsilon\max(1,|x|)$ is usual). Trusting a finite-difference check of a gradient without a tolerance that reflects these errors.

Numerical considerations

Complex-step differentiation, Richardson extrapolation and automatic differentiation (III.10) avoid or reduce the cancellation problem. When finite differences are used as a test of AD code, a centered difference with $h\approx10^{-5}$–$10^{-6}$ and a relative tolerance around $10^{-7}$ is a common, defensible choice for smooth functions.

Engineering and ML connections

Gradient checking in machine-learning code compares backpropagation with centered differences. Simulation codes estimate Jacobians by finite differences when no derivative code exists, and must choose step sizes per variable. The same truncation-versus-rounding trade-off appears whenever a continuous limit is replaced by a finite, floating-point computation.

Exercises

Summary and checkpoints

Forward and backward differences have truncation error of order $h$, centered differences of order $h^2$; rounding error grows like $\varepsilon/h$. Their sum is minimised near $h\approx\sqrt\varepsilon$ (forward) or $\sqrt[3]\varepsilon$ (centered), and tinier steps make results worse. Next, III.15 studies the matching question for integrals.