How do signed and squared histories set an adaptive update?

At the canonical start $(2,1)$, the gradient is $(2,4)$. Ordinary descent with rate $1/4$ would move by $(-1/2,-1)$. Adam’s first update instead moves both coordinates by nearly $-1/4$. Predict how dividing by the square root of a squared-gradient history produces that change, then inspect the moments rather than treating the arrow as a magic rescaling.

Intuition: direction history and magnitude history

A signed average remembers which way gradients have tended to point. An average of their squares remembers their typical magnitudes while discarding their signs. Adam uses the first history in the numerator and the square root of the second in the denominator, separately for each coordinate.

This can make coordinates with very different gradient magnitudes receive similarly sized first updates. Later updates also depend on the entire retained history, so a new sign change need not reverse the signed average immediately. Adam is not a Hessian method, a unit-gradient method, or a promise that every update lowers the objective.

Both moment vectors start at zero. Their initial geometric weights sum to less than one, which shrinks their early raw values. Bias correction removes this zero-initialization weight deficit. It does not replace a sampled gradient with the exact full gradient or make old gradients describe the current point perfectly.

Definition: Algorithm 1 with an explicit first time index

These elementwise equations follow Kingma and Ba’s Adam: A Method for Stochastic Optimization (2014), arXiv:1412.6980, Algorithm 1. Start with $m_0=v_0=0$. At time $t\geq1$, evaluate $g_t=\nabla L(\theta_{t-1})$ and use

Squares, square roots, and division are componentwise. Require $0\leq\beta_1,\beta_2<1$, $\eta>0$, and $\epsilon>0$. Epsilon is outside the square root. The first update uses $t=1$, corresponding to the stage’s first iteration $k=0$.

The lesson uses $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$, rate $\eta=0.25$, and the default quadratic start $(2,1)$. The rate is this toy lesson’s setting for visible movement. The notation $v$ now names a raw squared-gradient moment, distinct from the unnormalized velocity in the preceding momentum chapter.

Mechanics: account for every state component

Read the current parameters and gradient, previous moments, updated moments, corrected moments, time, decay factors, epsilon, and displacement together. The denominator for coordinate j is $\sqrt{\widehat v_{t,j}}+\epsilon$; it must use the same corrected second moment shown in the inspector.

The first time is 1 even though the history table begins at iteration 0. Increasing the replay cursor chooses an evaluated moment-and-parameter state; it does not change the time denominator independently or recompute a different gradient. Reset restores zero moments, zero initial time, the canonical point, and the canonical settings.

Change the rate or decay factors in the opening explorer to produce another run. The worked stage retains the canonical run. Epsilon remains the displayed configured positive constant; the deliberately enlarged epsilon in one exercise is a separate hand probe, not a claim about the explorer’s default.

Derivation: correct the missing weight, not the current gradient

Unrolling the first moment gives

Its weights sum to

Dividing by that sum produces a normalized weighted history. For a constant gradient g, $m_t=(1-\beta_1^t)g$ and $\widehat m_t=g$ exactly. If gradient distributions have a constant mean over time, the same calculation explains the initialization bias in expectation. When gradients change, the corrected vector remains a history average rather than necessarily the newest gradient.

Likewise,

This is a raw second moment: it averages squares without centering around a mean. It is not a variance, a covariance matrix, or a Hessian. Squaring removes each gradient sign; the first moment determines the update sign because the denominator is positive.

At the first step from zero history, $\widehat m_1=g_1$ and $\widehat v_1=g_1^{\odot2}$, so coordinate j has

For a nonzero component much larger than epsilon, this is close to $-\eta\operatorname{sign}(g_{1,j})$. For a zero component it is exactly zero. This first-step cancellation does not hold in the same form once historical gradients differ.

Worked example: two different magnitudes, nearly equal steps

On $L=x^2/2+2y^2$ at $\theta_0=(2,1)$, $g_1=(2,4)$. At $t=1$ the raw moments are

Their denominators are $1-0.9=0.1$ and $1-0.999=0.001$, giving exact hand values

With the lesson’s epsilon and rate, the update is

Therefore

The raw gradient’s y component was twice its x component, but its corrected squared magnitude produces twice the square-root denominator. This first update is different from plain descent’s $(3/2,0)$ at the same nominal rate. One step does not decide which algorithm is preferable for a complete problem.

As a separate scalar sign probe, prescribe gradients $2$ then $-2$ rather than taking the canonical quadratic’s next gradient. At $t=2$,

Bias denominators $0.19$ and $0.001999$ give $\widehat m_2=-2/19$ and $\widehat v_2=4$. The second moment retains magnitude while the signed moment reverses. This explicit gradient sequence tests the state recurrence; it is not a replacement for the actual gradients in the stage’s quadratic trajectory.

Executable reference code

The standard-library program implements Algorithm 1 with an independent 80-digit Decimal calculation. The first moments and corrected literal values are exact for the decimal inputs; later square roots and divisions are high-precision approximations. Assertions cover the canonical first update, positive epsilon, zero history, zero current input with retained history, sign reversals, and an enlarged-epsilon hand calculation.

It also evolves eight actual quadratic steps and independently checks the moment recurrences against explicit weighted gradient histories. The NumPy program evolves multiple starts with array operations and cross-checks those moments against a lower-triangular weight matrix acting on the saved gradients. Its binary64 tolerance is $10^{-14}$ on that bounded run. Neither program imports another book reference or installs an optimizer library.

"""Independent 80-digit Decimal Adam, Algorithm 1: https://arxiv.org/pdf/1412.6980.

The first moment step uses t=1; epsilon is outside the square root.
Decimal outputs involving square roots are high-precision approximations.
"""
from decimal import Decimal as D, localcontext


def adam_update(gradient, m=(0, 0), v=(0, 0), t=1,
                eta="0.25", beta1="0.9", beta2="0.999", epsilon="1e-8"):
    if type(t) is not int or t < 1:
        raise ValueError("The first update is t=1, not t=0")
    if any(len(vector) != 2 for vector in (gradient, m, v)):
        raise ValueError("Require two-component gradient and moment vectors")
    with localcontext() as context:
        context.prec = 80
        gradient, m, v = (tuple(D(str(x)) for x in vector) for vector in (gradient, m, v))
        eta, beta1, beta2, epsilon = map(lambda x: D(str(x)), (eta, beta1, beta2, epsilon))
        if not all(x.is_finite() for x in (*gradient, *m, *v, eta, beta1, beta2, epsilon)):
            raise ValueError("Require finite inputs")
        if eta <= 0 or epsilon <= 0 or not 0 <= beta1 < 1 or not 0 <= beta2 < 1 or any(x < 0 for x in v):
            raise ValueError("Require eta,epsilon>0, betas in [0,1), and nonnegative second moments")
        next_m = tuple(beta1*old+(1-beta1)*g for old, g in zip(m, gradient))
        next_v = tuple(beta2*old+(1-beta2)*g*g for old, g in zip(v, gradient))
        m_hat = tuple(value/(1-beta1**t) for value in next_m)
        v_hat = tuple(value/(1-beta2**t) for value in next_v)
        update = tuple(-eta*mean/(second.sqrt()+epsilon) for mean, second in zip(m_hat, v_hat))
        return {"t": t, "gradient": gradient, "m": next_m, "v": next_v,
                "m_hat": m_hat, "v_hat": v_hat, "update": update}


def main():
    with localcontext() as context:
        context.prec = 80
        first = adam_update((2, 4))
        assert first["m"] == (D("0.2"), D("0.4"))
        assert first["v"] == (D("0.004"), D("0.016"))
        assert first["m_hat"] == (2, 4) and first["v_hat"] == (4, 16)
        assert first["update"] == (-D("0.25")*2/(2+D("1e-8")), -D("0.25")*4/(4+D("1e-8")))
        zero = adam_update((0, 0))
        assert zero["m_hat"] == zero["v_hat"] == zero["update"] == (0, 0)
        retained = adam_update((0, 0), first["m"], first["v"], t=2)
        assert retained["m"] == (D("0.18"), D("0.36"))
        assert all(x < 0 for x in retained["update"])
        reversed_gradient = adam_update((-2, -4), first["m"], first["v"], t=2)
        assert reversed_gradient["m"] == (D("-0.02"), D("-0.04"))
        assert reversed_gradient["v"] == (D("0.007996"), D("0.031984"))
        assert reversed_gradient["m_hat"] == (-D(2)/19, -D(4)/19)
        assert reversed_gradient["v_hat"] == (4, 16)
        small_reversal = adam_update(("-0.1", 0), first["m"], first["v"], t=2)
        assert small_reversal["m"] == (D("0.17"), D("0.36"))
        assert small_reversal["update"][0] < 0  # Memory still points against the new negative gradient.
        large_epsilon = adam_update((2, 4), epsilon=1)
        assert large_epsilon["update"] == (-D(1)/6, -D(1)/5)
        theta, m, v, gradients, path = (D(2), D(1)), (D(0), D(0)), (D(0), D(0)), [], []
        for t in range(1, 9):
            gradient = (theta[0], 4*theta[1])
            gradients.append(gradient)
            item = adam_update(gradient, m, v, t)
            weighted_m = tuple(D("0.1")*sum(D("0.9")**(t-i)*g[j] for i, g in enumerate(gradients, 1)) for j in (0, 1))
            weighted_v = tuple(D("0.001")*sum(D("0.999")**(t-i)*g[j]*g[j] for i, g in enumerate(gradients, 1)) for j in (0, 1))
            assert all(abs(actual-expected) < D("1e-70") for actual, expected in zip(item["m"], weighted_m))
            assert all(abs(actual-expected) < D("1e-70") for actual, expected in zip(item["v"], weighted_v))
            theta = tuple(theta[j]+item["update"][j] for j in (0, 1))
            m, v = item["m"], item["v"]
            path.append(theta)
        for kwargs in ({"t": 0}, {"beta1": 1}, {"beta2": -1}, {"epsilon": 0}, {"eta": 0}):
            try:
                adam_update((2, 4), **kwargs)
            except ValueError:
                pass
            else:
                raise AssertionError("Invalid Adam configuration accepted")
        print("First-step high-precision moments and update:", first)
        print("First next theta, 80-digit approximations:", path[0])
        print("Zero current gradient can retain a nonzero update from moment history.")
        print("A finite Adam run does not certify convergence or global optimality.")


if __name__ == "__main__":
    main()

"""Vectorized binary64 Adam paths, checked against explicit weighted gradient histories."""
import numpy as np


def trajectories(starts, iterations=8, eta=0.25, beta1=0.9, beta2=0.999, epsilon=1e-8):
    starts = np.asarray(starts, dtype=np.float64)
    if starts.ndim != 2 or starts.shape[1] != 2 or not np.all(np.isfinite(starts)):
        raise ValueError("Require finite starts of shape (n,2)")
    if not np.all(np.isfinite([eta, beta1, beta2, epsilon])) or eta <= 0 or epsilon <= 0 or not 0 <= beta1 < 1 or not 0 <= beta2 < 1:
        raise ValueError("Require finite parameters, eta,epsilon>0, and betas in [0,1)")
    if type(iterations) is not int or not 1 <= iterations <= 1000:
        raise ValueError("Require an integer iteration count in [1,1000]")
    path = np.empty((iterations+1, len(starts), 2), dtype=np.float64)
    gradients = np.empty((iterations, len(starts), 2), dtype=np.float64)
    moments, seconds, updates = np.empty_like(gradients), np.empty_like(gradients), np.empty_like(gradients)
    m, v = np.zeros_like(starts), np.zeros_like(starts)
    path[0] = starts
    for k in range(iterations):
        t = k+1
        gradients[k] = path[k]*np.array([1.0, 4.0])
        m = beta1*m+(1-beta1)*gradients[k]
        v = beta2*v+(1-beta2)*gradients[k]**2
        moments[k], seconds[k] = m, v
        m_hat, v_hat = m/(1-beta1**t), v/(1-beta2**t)
        updates[k] = -eta*m_hat/(np.sqrt(v_hat)+epsilon)
        path[k+1] = path[k]+updates[k]
        if not np.all(np.isfinite(path[k+1])) or not np.all(np.isfinite(m)) or not np.all(np.isfinite(v)):
            raise OverflowError(f"Non-finite binary64 Adam state at t={t}")
    return path, gradients, moments, seconds, updates


def weighted_moments(gradients, beta):
    count = len(gradients)
    lag = np.arange(count)[:, None]-np.arange(count)[None, :]
    weights = np.where(lag >= 0, (1-beta)*beta**np.maximum(lag, 0), 0.0)
    return np.einsum("ti,inj->tnj", weights, gradients)


def main():
    path, gradients, moments, seconds, updates = trajectories([[2, 1], [-2, -1], [0, 0]])
    np.testing.assert_allclose(moments, weighted_moments(gradients, 0.9), rtol=0, atol=1e-14)
    np.testing.assert_allclose(seconds, weighted_moments(gradients**2, 0.999), rtol=0, atol=1e-14)
    np.testing.assert_allclose(moments[0, 0], [0.2, 0.4], rtol=0, atol=1e-14)
    np.testing.assert_allclose(seconds[0, 0], [0.004, 0.016], rtol=0, atol=1e-14)
    expected_first = -0.25*np.array([2.0, 4.0])/(np.array([2.0, 4.0])+1e-8)
    np.testing.assert_allclose(updates[0, 0], expected_first, rtol=0, atol=1e-14)
    np.testing.assert_allclose(path[1, 0], [1.75000000125, 0.750000000625], rtol=0, atol=1e-14)
    assert np.all(path[:, 2] == 0) and np.all(updates[:, 2] == 0)
    probes = np.array([[[2.0, 4.0]], [[0.0, 0.0]], [[-2.0, -4.0]]])
    probe_m, probe_v = weighted_moments(probes, 0.9), weighted_moments(probes**2, 0.999)
    t = np.arange(1, 4)[:, None, None]
    probe_updates = -0.25*(probe_m/(1-0.9**t))/(np.sqrt(probe_v/(1-0.999**t))+1e-8)
    assert np.all(probe_updates[1] < 0) and np.all(probe_updates[2] > 0)
    np.testing.assert_allclose(probe_m[1, 0], [0.18, 0.36], rtol=0, atol=1e-14)
    reversal = weighted_moments(np.array([[[2.0, 4.0]], [[-2.0, -4.0]]]), 0.9)
    np.testing.assert_allclose(reversal[1, 0]/(1-0.9**2), [-2/19, -4/19], rtol=0, atol=1e-14)
    large_epsilon = trajectories([[2, 1]], iterations=1, epsilon=1)[4]
    np.testing.assert_allclose(large_epsilon[0, 0], [-1/6, -1/5], rtol=0, atol=1e-14)
    for kwargs in ({"epsilon": 0}, {"beta1": 1}, {"beta2": np.nan}, {"eta": 0}):
        try:
            trajectories([[2, 1]], **kwargs)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid Adam configuration accepted")
    print("Binary64 Adam path for the first start ≈", path[:, 0])
    print("Binary64 first moments ≈", moments[:, 0])
    print("Weighted-history cross-check: absolute tolerance 1e-14 on this bounded run.")
    print("The raw second moment is not a Hessian or a centered variance.")


if __name__ == "__main__":
    main()

Explore: zero coordinates and changing direction

Choose zero gradients. With zero moment history and a zero initial gradient, both raw and corrected moments remain zero. The denominator is epsilon, so the displacement is the defined zero vector. Choose one zero component instead: the corresponding history stays zero while the other coordinate evolves.

Use the sign-change preset and compare current gradient signs with corrected first-moment signs before reading the displacement. The squared moment remains nonnegative; it cannot reverse a denominator’s sign. A small opposing gradient can leave the signed average pointing its old way.

For a hand example of that lag, prescribe scalar gradients $2$ then $-1/10$ with $\beta_1=9/10$. The second first moment is $0.9(0.2)+0.1(-0.1)=0.17>0$. The update is therefore still negative despite a negative current gradient. With retained nonzero moments, a zero current gradient can likewise leave a nonzero update after decay. A fresh zero-history preset and a zero gradient inside an ongoing run are different states.

Common mistakes and counterexamples

Starting the correction at $t=0$ divides by zero. Using $1-\beta$ at every time treats all updates like the first one. Bias correction normalizes a whole weighted history, so its denominator depends on the actual count of parameter updates, not the number of examples in a mini-batch.

Moving epsilon inside the root changes the algorithm. To see the difference exactly, keep the first corrected moments but use $\epsilon=1$ for a hand probe. The denominators are $\sqrt4+1=3$ and $\sqrt{16}+1=5$, so the update is $(-1/6,-1/5)$ and next point $(11/6,4/5)$. Replacing the denominators by $\sqrt5$ and $\sqrt{17}$ would produce different values.

Numerical considerations

The stage records binary64 approximations, including powers, corrections, square roots, and parameter additions. The Decimal reference is a high-precision cross-check rather than a symbolic guarantee that every returned decimal expansion is exact. Keep hand-exact moments separate from rounded numerical trajectory values.

When a decay is close to 1, calculating $1-\beta^t$ subtracts nearby numbers and can lose relative accuracy. Squaring a very small gradient can underflow; squaring a large finite gradient can overflow before a parameter update is computed. Check raw and corrected moments for finite values instead of accepting a misleading zero update from an infinite denominator.

Positive epsilon prevents division by zero at zero corrected second moment and controls behavior when gradient scale is tiny. It also means rescaling all gradients does not give exact scale invariance in general: the constant epsilon does not scale with them. Epsilon placement and magnitude are therefore mathematical parts of the chosen update, not display details.

Engineering and ML connections

Momentum retains an unnormalized accumulated gradient, while Adam retains a normalized signed average plus a normalized squared-gradient history and corrects their initial weights. Stochastic mini-batches can supply the gradients, but Adam’s moments do not turn biased sampling into unbiased sampling.

A restart must preserve parameters, both moment vectors, the time count, and the configured decay and epsilon values to continue the same run. Keeping only the parameters and resetting the moments changes later displacements. Automatic differentiation computes gradients; Adam’s state transition is the separate mechanism using them. Convexity describes an objective’s global structure, which no optimizer name can supply automatically.

Exercises

Use exact rational answers for moment arithmetic and the enlarged-epsilon probe. State when a gradient sequence is prescribed rather than generated by the canonical objective.

Summary and checkpoints

Adam updates signed and squared-gradient histories, divides by their accumulated weight sums, and scales each coordinate by a square-root denominator with epsilon outside. Check that you can recover the canonical first moments and update, identify the first time as 1, preserve sign information in the correct vector, and distinguish zero input from zero history. Adaptive state explains movement; it does not by itself prove convergence or optimality.