Which way is uphill — and how steep is every other way?

At $(1,\tfrac12)$ on the valley $x^2/2+2y^2$, the gradient is $(1,2)$. In the direction $(1,0)$ the height changes at rate $1$. Choose Along the gradient: the rate rises to $\|\nabla f\|=\sqrt5\approx2.236$, the largest possible. Choose Along the level curve: the rate is exactly $0$.

Intuition: one vector answers every direction

Walking from a point in a direction $\mathbf u$, the height changes at some rate. You might expect a different calculation for every direction. For a differentiable function there is a single vector — the gradient — and the rate in direction $\mathbf u$ is just its component along $\mathbf u$. The profile plot shows the consequence: as the direction turns, the rate traces a cosine wave.

Definition

The gradient of $f$ at $p$ is the vector of partial derivatives,

For a unit vector $\mathbf u$, the directional derivative is

When $f$ is differentiable, $D_{\mathbf u}f=\nabla f\cdot\mathbf u$. A nonzero direction vector $\mathbf d$ is normalised first: $\mathbf u=\mathbf d/\|\mathbf d\|$. The zero vector has no direction.

Mechanics: what the explorer computes

The gradient is exact at rational points. The unnormalised product $\nabla f\cdot\mathbf d$ is exact; dividing by $\|\mathbf d\|$ is exact when $\|\mathbf d\|^2$ is the square of a rational and is otherwise shown with $\approx$. The gradient arrow is drawn at the scale stated in the title, so its direction is faithful and its length is proportional. The level-curve tangent is perpendicular to the gradient, and the inspector checks $\nabla f\cdot\mathbf t=0$ exactly.

Derivation: why the dot product appears

Differentiability means $f(p+\boldsymbol\delta)=f(p)+\nabla f\cdot\boldsymbol\delta+o(\|\boldsymbol\delta\|)$. Put $\boldsymbol\delta=h\mathbf u$ and divide by $h$:

Letting $h\to0$ gives $D_{\mathbf u}f=\nabla f\cdot\mathbf u=\|\nabla f\|\cos\theta$, where $\theta$ is the angle between $\mathbf u$ and $\nabla f$. The rate is largest when $\theta=0$ (steepest ascent, value $\|\nabla f\|$), smallest when $\theta=\pi$, and zero when $\mathbf u\perp\nabla f$. Along a level curve $f$ is constant, so its tangent direction has zero rate: the gradient is perpendicular to regular level curves.

Worked example

For $f=x^2/2+2y^2$ at $(1,\tfrac12)$: $\nabla f=(x,4y)=(1,2)$. With $\mathbf d=(3,4)$, $\|\mathbf d\|=5$, so

The steepest rate is $\sqrt{1+4}=\sqrt5$, and the tangent to the level curve is $\mathbf t=(-2,1)$, for which $\nabla f\cdot\mathbf t=0$.

Stationary points and undefined cases

Where $\nabla f=\mathbf 0$ (the Stationary point preset), every directional derivative is $0$ and there is no direction of steepest ascent; the stage draws no gradient arrow rather than an arbitrary one. A zero direction vector cannot be normalised, so the directional derivative is reported as undefined. At points where $f$ is not differentiable, the formula $\nabla f\cdot\mathbf u$ is not guaranteed even when the partials exist; the surfaces here are polynomials, which are differentiable everywhere.

Executable reference code

The reference computes exact gradients and directional derivatives with Fraction, and confirms $D_{\mathbf u}f$ against the limit definition along rational unit directions such as $(3/5,4/5)$. The NumPy version maximises the directional derivative over many angles and finds the gradient direction.

"""Gradient and directional derivatives, exact; the limit definition agrees with grad . u (III.8)."""
from fractions import Fraction as F

f = lambda x, y: x * x / 2 + 2 * y * y
grad = lambda x, y: (x, 4 * y)

p = (F(1), F(1, 2))
g = grad(*p)
assert g == (1, 2)


def directional(d):
    if d == (0, 0):
        raise ValueError("the zero vector has no direction")
    n2 = d[0] ** 2 + d[1] ** 2
    return (g[0] * d[0] + g[1] * d[1]), n2           # rate = numerator / sqrt(n2)


num, n2 = directional((F(3), F(4)))
assert (num, n2) == (11, 25)                          # rate 11/5

# Limit definition along the unit direction u = (3/5, 4/5): m(h) is a polynomial of degree 1 in h.
u = (F(3, 5), F(4, 5))
m = lambda h: (f(p[0] + h * u[0], p[1] + h * u[1]) - f(*p)) / h
assert 2 * m(F(1, 4)) - m(F(1, 2)) == F(11, 5)

# The gradient is perpendicular to the level-curve tangent t = (-g_y, g_x).
t = (-g[1], g[0])
assert g[0] * t[0] + g[1] * t[1] == 0

"""The steepest direction maximises grad . u over all unit vectors (III.8)."""
import numpy as np

g = np.array([1.0, 2.0])
theta = np.linspace(-np.pi, np.pi, 3601)
U = np.stack([np.cos(theta), np.sin(theta)], axis=1)
rates = U @ g
best = U[np.argmax(rates)]
assert abs(rates.max() - np.linalg.norm(g)) < 1e-6
assert np.allclose(best, g / np.linalg.norm(g), atol=1e-3)
assert abs(rates.min() + np.linalg.norm(g)) < 1e-6

Explore

On Saddle, find two directions with rate zero: they are the two arms of the level curve through the point. On Tilted plane, the gradient is the same everywhere and the contours are parallel lines perpendicular to it. Shorten $\mathbf d$ without changing its direction: the directional derivative does not change, because $\mathbf d$ is normalised.

Common mistakes

Using an unnormalised direction, which scales the “rate” by $\|\mathbf d\|$. Drawing the gradient tangent to the contour instead of perpendicular to it. Assuming the gradient points toward a minimum: it points toward the steepest increase; the steepest decrease is $-\nabla f$.

Numerical considerations

Normalising a tiny direction vector divides by a tiny number and amplifies rounding; the zero vector must be rejected explicitly, not left to produce NaN. When $\nabla f$ is nearly zero its direction is numerically unreliable even if its components are accurate: steepest-ascent directions near stationary points are ill-conditioned.

Engineering and ML connections

Gradient descent (VI.4) steps along $-\nabla f$ because it is locally the steepest decrease. Directional derivatives appear in line searches and in Jacobian–vector products, which automatic differentiation (III.10) computes without forming the whole gradient. Level-set geometry also underlies constraints in optimisation (VI.9).

Exercises

Summary and checkpoints

The gradient collects the partial derivatives; for a unit direction $\mathbf u$, $D_{\mathbf u}f=\nabla f\cdot\mathbf u$. It points along steepest increase with rate $\|\nabla f\|$ and is perpendicular to regular level curves. At stationary points it vanishes, and zero directions have no rate. Next, III.9 differentiates maps with several outputs: the Jacobian.