What changes when the optimizer remembers its direction?

Start at $(2,1)$ with zero velocity, $\beta=0.8$, and $\eta=0.25$. The first point matches ordinary gradient descent: $(3/2,0)$. Before advancing again, predict whether zero current y gradient also means zero next y displacement. The stored velocity supplies the missing information.

Intuition: current slope plus a fading history

Plain gradient descent chooses a displacement from the current gradient alone. Momentum adds memory. Gradients that repeatedly point the same way accumulate, while older contributions fade. A newly opposing gradient must first cancel some retained history before the accumulated direction can reverse.

The mechanical analogy is motion with damping, but the velocity here is an optimizer state vector with a stated sign convention. Positive velocity produces a negative parameter displacement because the update subtracts it. It need not equal physical velocity, the current gradient, or a unit direction.

Memory can help persistent directions dominate short-lived reversals, but it can also carry a point past a low-loss region. Inspect the extra state instead of assuming that smoothing or convexity makes every step lower the objective.

Definition: the complete two-state recurrence

This chapter uses unnormalized accumulated gradients:

The parameters satisfy $\eta>0$ and $0\leq\beta<1$. Both $\theta_k$ and $v_k$ belong to the optimizer state. Two runs reaching the same parameter point with different velocities can take different next steps.

The recurring quadratic is $L=a(x-c)^2/2+2y^2$, with gradient $(a(x-c),4y)$. The canonical run uses $a=1,c=0$, start $(2,1)$, $\eta=1/4$, and $\beta=4/5$. A normalized moving average $\beta v+(1-\beta)g$ is a different convention; the gradient factor is deliberately absent here.

Mechanics: inspect the retained direction before adding it

The selected step exposes the previous velocity, current gradient, updated velocity, decay, rate, displacement, and next parameters. First compute $\beta v_k$, then add $g_k$, then multiply the resulting vector by $-\eta$. The arrow’s endpoint difference must equal that last vector, not merely $-\eta g_k$.

Reset restores the canonical parameters and zero velocity. Changing the rate, start, or decay creates a new evaluated trajectory. Replay selects stored parameter-and-velocity states; it does not reconstruct velocity by guessing from a rounded path or reverse a floating-point update.

The no-memory preset sets $\beta=0$. It should recover ordinary gradient descent with the same point and rate. Compare it with the canonical momentum run at the second step, where their histories first produce different displacements.

Derivation: what the memory actually contains

Unroll the velocity recurrence from $v_0=0$:

Every previous gradient is weighted by a power of the decay factor. For a constant gradient g, this becomes

At $\beta=4/5$, persistent gradient magnitude is amplified toward five times g under this convention. The normalized average instead tends to g. For identical histories from zero state, its velocity is $(1-\beta)$ times the unnormalized one; matching displacements would require dividing its learning rate by $1-\beta$.

The parameter history gives another view. Since $v_k=(\theta_{k-1}-\theta_k)/\eta$ for $k\geq1$,

For a quadratic error mode of curvature $\lambda$, this is

It is a second-order recurrence, unlike gradient descent’s single error multiplier. The previous rate condition for plain descent is therefore not a complete momentum stability analysis. The memory and rate act together.

Worked example: the same first step, a different second step

At $\theta_0=(2,1)$, $g_0=(2,4)$ and $v_0=(0,0)$, so

Now recompute $g_1=(3/2,0)$, retaining $v_1$. The next velocity is

Its displacement and next point are

The new loss is

The loss rises from $1.125$ to $1.5428125$ in exact decimal arithmetic. The y contribution rises because the retained positive y velocity moves the parameter below zero. All equalities here are hand calculations; the evaluated stage history is labeled binary64 approximation.

Executable reference code

The standard-library reference records exact Fraction frames and verifies the literal first and second steps, including the rising loss $4937/3200$. It then checks the same path with the independently derived second-order parameter recurrence. Additional assertions cover no memory, constant-gradient accumulation, and a zero current gradient with retained velocity.

The NumPy program advances several starts simultaneously. It independently builds a four-state linear transition matrix for quadratic errors and velocities, then cross-checks the trajectory with numpy.linalg.matrix_power. This library calculation verifies the bounded implementation; it does not create a universal guarantee for arbitrary momentum settings or objectives.

"""Exact unnormalized momentum: v_next=beta*v+g, theta_next=theta-eta*v_next."""
from fractions import Fraction as F


def step(theta, velocity=(0, 0), eta=F(1, 4), beta=F(4, 5), a=1, c=0):
    x, y = map(F, theta)
    vx, vy = map(F, velocity)
    eta, beta, a, c = map(F, (eta, beta, a, c))
    if eta <= 0 or a <= 0 or not 0 <= beta < 1:
        raise ValueError("Require eta>0, a>0, and 0<=beta<1")
    gradient = (a*(x-c), 4*y)
    next_velocity = (beta*vx+gradient[0], beta*vy+gradient[1])
    update = tuple(-eta*v for v in next_velocity)
    next_theta = (x+update[0], y+update[1])
    return {
        "theta": (x, y), "loss": a*(x-c)**2/2+2*y*y,
        "velocity": (vx, vy), "gradient": gradient, "next_velocity": next_velocity,
        "update": update, "next_theta": next_theta,
        "next_loss": a*(next_theta[0]-c)**2/2+2*next_theta[1]**2,
    }


def trajectory(start=(2, 1), iterations=8, eta=F(1, 4), beta=F(4, 5), a=1, c=0):
    if type(iterations) is not int or not 0 <= iterations <= 1000:
        raise ValueError("Require an integer iteration count in [0,1000]")
    eta, beta, a, c = map(F, (eta, beta, a, c))
    if eta <= 0 or a <= 0 or not 0 <= beta < 1:
        raise ValueError("Require eta>0, a>0, and 0<=beta<1")
    theta, velocity, frames = tuple(map(F, start)), (F(0), F(0)), []
    for _ in range(iterations):
        item = step(theta, velocity, eta, beta, a, c)
        frames.append(item)
        theta, velocity = item["next_theta"], item["next_velocity"]
    return frames


def main():
    frames = trajectory()
    first, second = frames[:2]
    assert first["next_velocity"] == (2, 4)
    assert first["next_theta"] == (F(3, 2), 0) and first["next_loss"] == F(9, 8)
    assert second["gradient"] == (F(3, 2), 0)
    assert second["next_velocity"] == (F(31, 10), F(16, 5))
    assert second["update"] == (F(-31, 40), F(-4, 5))
    assert second["next_theta"] == (F(29, 40), F(-4, 5))
    assert second["next_loss"] == F(4937, 3200) > second["loss"]
    # Eliminate velocity to check a different, second-order parameter recurrence.
    for k in range(1, len(frames)):
        previous, current = frames[k-1]["theta"], frames[k]["theta"]
        expected = tuple((1+F(4, 5)-F(1, 4)*lam)*now-F(4, 5)*old
                         for now, old, lam in zip(current, previous, (1, 4)))
        assert frames[k]["next_theta"] == expected
    residual = step((0, 0), (1, -2))
    assert residual["gradient"] == (0, 0)
    assert residual["next_velocity"] == (F(4, 5), F(-8, 5))
    assert residual["next_theta"] == (F(-1, 5), F(2, 5))
    no_memory = step((2, 1), (99, -99), beta=0)
    assert no_memory["next_velocity"] == (2, 4) and no_memory["next_theta"] == first["next_theta"]
    constant_velocity = F(0)
    for n in range(1, 5):
        constant_velocity = F(4, 5)*constant_velocity+1
        assert constant_velocity == (1-F(4, 5)**n)/(1-F(4, 5))
    for invalid in (-1, 1):
        try:
            step((2, 1), beta=invalid)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid momentum coefficient accepted")
    print("Exact first frame:", first)
    print("Exact second frame:", second)
    print("Momentum can keep moving at zero current gradient and can increase loss on a step.")


if __name__ == "__main__":
    main()

"""Binary64 batch momentum, cross-checked with powers of a four-state transition matrix."""
import numpy as np


def trajectories(starts, iterations=8, eta=0.25, beta=0.8, a=1.0, c=0.0):
    starts = np.asarray(starts, dtype=np.float64)
    if starts.ndim != 2 or starts.shape[1] != 2 or not np.all(np.isfinite(starts)):
        raise ValueError("Require finite starts of shape (n,2)")
    if not np.all(np.isfinite([eta, beta, a, c])) or eta <= 0 or a <= 0 or not 0 <= beta < 1:
        raise ValueError("Require finite parameters, eta>0, a>0, 0<=beta<1")
    if type(iterations) is not int or not 0 <= iterations <= 1000:
        raise ValueError("Require an integer iteration count in [0,1000]")
    center, hessian = np.array([c, 0.0]), np.diag([a, 4.0])
    path = np.empty((iterations+1, len(starts), 2), dtype=np.float64)
    velocity = np.zeros_like(path)
    path[0] = starts
    for k in range(iterations):
        gradient = (path[k]-center) @ hessian
        velocity[k+1] = beta*velocity[k]+gradient
        path[k+1] = path[k]-eta*velocity[k+1]
        if not np.all(np.isfinite(path[k+1])) or not np.all(np.isfinite(velocity[k+1])):
            raise OverflowError(f"Non-finite binary64 state at step {k+1}")
    losses = np.einsum("kni,ij,knj->kn", path-center, hessian, path-center)/2
    if not np.all(np.isfinite(losses)):
        raise OverflowError("Non-finite binary64 loss")
    return path, velocity, losses


def main():
    starts = np.array([[2, 1], [-2, -1], [0, 0]], dtype=np.float64)
    path, velocity, losses = trajectories(starts)
    h, identity = np.diag([1.0, 4.0]), np.eye(2)
    transition = np.block([[identity-0.25*h, -0.25*0.8*identity], [h, 0.8*identity]])
    initial = np.column_stack((starts, np.zeros_like(starts)))
    for k in range(9):
        direct = initial @ np.linalg.matrix_power(transition, k).T
        np.testing.assert_allclose(path[k], direct[:, :2], rtol=0, atol=1e-12)
        np.testing.assert_allclose(velocity[k], direct[:, 2:], rtol=0, atol=1e-12)
    np.testing.assert_allclose(velocity[2, 0], [3.1, 3.2], rtol=0, atol=1e-14)
    np.testing.assert_allclose(path[2, 0], [0.725, -0.8], rtol=0, atol=1e-14)
    np.testing.assert_allclose(losses[:3, 0], [4, 1.125, 1.5428125], rtol=0, atol=1e-14)
    assert losses[2, 0] > losses[1, 0]
    assert np.all(path[:, 2] == 0) and np.all(velocity[:, 2] == 0)
    plain, _, _ = trajectories([[2, 1]], iterations=2, beta=0)
    np.testing.assert_array_equal(plain[:, 0], [[2, 1], [1.5, 0], [1.125, 0]])
    print("Binary64 momentum path for the first start ≈", path[:, 0])
    print("Binary64 velocities ≈", velocity[:, 0])
    print("Matrix-power cross-check: absolute tolerance 1e-12 on this bounded run.")


if __name__ == "__main__":
    main()

Explore: compare damping, history, and displacement

Choose no memory and verify that the second point is $(9/8,0)$ instead of $(29/40,-4/5)$. The parameter point at the start of the second step is the same, but the velocity rule is different. Return to the canonical decay and locate the first y sign reversal; compare the current gradient with the accumulated direction before each move.

Reduce the decay, keeping the rate and start fixed. Older gradients now lose influence faster. Predict which terms in $v_{k+1}=\sum_j\beta^{k-j}g_j$ change before judging the plotted path. Increasing decay also changes accumulated magnitude under this unnormalized convention, so it is not solely a change in smoothness.

As a hand probe, suppose $\theta=(0,0)$ and retained velocity is $(1,-2)$. The current gradient is zero, but the next velocity is $(4/5,-8/5)$ and next point $(-1/5,2/5)$. This is a state-level counterexample to treating a zero current gradient as an automatic momentum stop. The explorer’s fresh runs always begin with zero velocity; this probe deliberately supplies nonzero history.

Common mistakes and counterexamples

Do not insert $(1-\beta)$ into this chapter’s recurrence. The normalized-average formula can be useful, but mixing it with an unnormalized learning rate changes the displacement. Likewise, multiplying the old velocity by the rate before updating it changes the chosen state convention unless the entire recurrence is transformed consistently.

A zero current gradient does not erase memory. A current gradient sign change need not immediately reverse the updated velocity. A convex objective still permits a momentum loss increase: its whole-domain minimum exists independently of whether this two-state algorithm descends on every step.

Numerical considerations

The exact reference uses rational inputs and arithmetic. Its binary64 counterpart rounds velocity accumulation and parameter addition; decimal values such as $3.1$ are not exactly representable in binary64. The inspector therefore shows approximation signs even when a hand calculation has a terminating decimal.

Large retained velocities can overflow or create a non-finite displacement before the objective is evaluated at the next point. Check the optimizer state as well as the loss. When $\beta$ is close to 1, a long persistent-gradient history has large total weight $1/(1-\beta)$, making scale and rate choices consequential.

Reproducible replay must preserve the stored velocity together with each parameter point. Reconstructing it from rounded coordinates can alter later updates. The matrix-power cross-check uses absolute tolerance $10^{-12}$ for the bounded run; this tolerance verifies arithmetic agreement, not convergence for all possible histories.

Engineering and ML connections

The velocity recurrence resembles a first-order filter of a gradient signal: it discounts older samples while retaining direction. In learning systems, keeping optimizer state across updates affects behavior; restarting only the parameters and silently discarding velocity creates a different run.

Learning-rate behavior explains how curvature acts on a one-state method. Momentum changes that recurrence, while stochastic gradients will add sampling variation. Adam also retains a signed first moment, but uses a normalized average, a squared-gradient moment, and bias correction. Similar names or arrows should not hide these different equations.

Exercises

Keep previous velocity, updated velocity, and displacement separate. Verify the loss after a step rather than assuming it must fall.

Summary and checkpoints

Momentum updates a decaying accumulated gradient before moving the parameters. Check that you can recover the first two canonical velocities and points, explain why the second y move persists, and account for the loss increase to $4937/3200$. The optimizer’s memory is mathematical state; a point and a current gradient alone no longer determine its next displacement.