When does a larger step help, oscillate, or run away?

At the default $\eta=1/4$, predict what happens separately to x and y. Now choose the no-decay preset $\eta=1/2$. The first loss still drops, but does the entire parameter vector converge to the minimum? Read the y values over several iterations before answering.

Intuition: one distance scale, two sensitivities

A narrow valley has one direction in which small movements change the gradient sharply and another in which they change it slowly. A shared learning rate must handle both. Too small a rate makes progress slow in the gentle direction; too large a rate overshoots in the steep one.

Overshooting does not automatically mean failure. A coordinate can cross its target on every step and still get closer. The useful distinction is whether its error magnitude shrinks, remains constant, or grows. A path’s zigzag shape alone cannot decide that distinction, and a short initial loss decrease cannot prove long-run convergence.

Definition: error contraction and all-start stability

For $L=a(x-c)^2/2+2y^2$ with $a>0$, the unique minimizer is $\theta^*=(c,0)$. Define error coordinates $e_x=x-c$ and $e_y=y$. Gradient descent uses $\theta_{k+1}=\theta_k-\eta\nabla L(\theta_k)$ with $\eta>0$.

A scalar error mode with curvature $\lambda>0$ obeys

Here the Hessian eigenvalues are $a$ and $4$, so the two multipliers are $r_x=1-\eta a$ and $r_y=1-4\eta$. In exact real arithmetic, convergence to the minimizer from every starting point requires

This is an all-start guarantee for this positive-curvature quadratic. An unstable or nondecaying mode whose initial error is exactly zero remains absent. Such a particular run can converge outside the all-start interval. Neither the formula nor a successful run supplies a universal rate bound for other objectives.

Mechanics: change the rate, keep the comparison controlled

Reset restores the default quadratic, $a=1,c=0$, start $(2,1)$, and rate $1/4$. The selected iteration exposes its current point, loss, gradient, rate, displacement, and next point. The error-factor rows connect that evaluated step with the recurrence’s longer-term prediction.

Compare the presets at $\eta=0.03,0.25,0.4,0.5,0.75$. Their x factors are $0.97,0.75,0.6,0.5,0.25$, while their y factors are $0.88,0,-0.6,-1,-2$. The first pair is slow, the middle negative factor produces decaying oscillation, and the final two y factors separate nondecay from growth.

The history stores evaluated binary64 states. Replay selects a previous state rather than undoing arithmetic. When the path leaves the plotting window, use the retained table and stated stopping reason; leaving the visible rectangle does not change the mathematical recurrence.

Derivation: the curvature condition is a geometric-series condition

The gradient is $(a(x-c),4y)$, so direct substitution gives

Unrolling each recurrence yields

For a nonzero initial component, a power $r^k$ tends to zero exactly when $\abs r<1$. Solve $-1<1-\eta\lambda<1$: subtract 1 and divide by positive $\lambda$ to obtain $0<\eta<2/\lambda$. Both modes must satisfy this, so the larger curvature sets the shared upper boundary.

The loss follows squared factors:

Within the strict all-start interval, each active contribution decreases. Negative factors can therefore produce decreasing loss even while coordinates alternate. At a boundary, an active contribution can remain fixed while another decreases.

Worked examples: a drop, a boundary, and an exception

With $\eta=1/4$, the factors are $(3/4,0)$. The canonical first step is

The second point is $(9/8,0)$ with loss $81/128$. For every $k\geq1$, the y component is zero and $x_k=2(3/4)^k$. This is the default worked replay.

At $\eta=1/2$, the factors become $(1/2,-1)$. The first two next points and losses are

Loss approaches $2$ while y alternates between $1$ and $-1$. The vector does not converge to $(0,0)$; decreasing loss has concealed a nondecaying mode.

At $\eta=3/4$, a start $(2,1)$ has $y_k=(-2)^k$, so its y error grows. A start $(2,0)$ instead has $y_k=0$ and $x_k=2(1/4)^k$, so it converges. This zero-mode exception does not refute the all-start boundary $1/2$.

For a balanced comparison inside the interval, $\eta=2/5$ gives factors $(3/5,-3/5)$. Its first point is $(6/5,-3/5)$ and loss $36/25$. Both squared contributions multiply by $9/25$ thereafter. This first loss is higher than the canonical first loss, yet its worst error factor is smaller: fastest initial improvement and fastest eventual contraction need not select the same rate.

Executable reference code

The standard-library program uses exact Fraction arithmetic for the gradient steps, then independently checks every stored point against the closed-form powers. Literal assertions cover slow contraction, canonical steps, the boundary, damped oscillation, a growing mode, and a zero-mode exception. It also supports positive curvature beyond the bounded explorer, including the hand-worked $a=8$ exercise.

The NumPy program advances multiple rates simultaneously with matrix operations, then compares the complete bounded path with a separately broadcasted power grid. Its documented tolerances apply to these binary64 calculations; the analytic recurrence establishes the convergence condition. No finite iteration budget proves an asymptotic theorem by itself.

"""Exact quadratic error factors and rate-dependent trajectories; no third-party imports."""
from fractions import Fraction as F


def factors(eta, a=1):
    eta, a = F(eta), F(a)
    if eta <= 0 or a <= 0:
        raise ValueError("Require positive curvature a and learning rate eta")
    return (1-eta*a, 1-4*eta)


def trajectory(start=(2, 1), eta=F(1, 4), iterations=8, a=1, c=0):
    if type(iterations) is not int or not 0 <= iterations <= 1000:
        raise ValueError("Require an integer iteration count in [0,1000]")
    x, y = map(F, start)
    eta, a, c = F(eta), F(a), F(c)
    fx, fy = factors(eta, a)
    path = [(x, y)]
    for _ in range(iterations):
        gradient = (a*(x-c), 4*y)
        x, y = x-eta*gradient[0], y-eta*gradient[1]
        path.append((x, y))
    for k, point in enumerate(path):
        assert point == (c+(F(start[0])-c)*fx**k, F(start[1])*fy**k)
    losses = [a*(x-c)**2/2+2*y*y for x, y in path]
    return path, losses


def main():
    assert factors(F(1, 10)) == (F(9, 10), F(3, 5))
    path, losses = trajectory(eta=F(1, 10), iterations=1)
    assert path[1] == (F(9, 5), F(3, 5)) and losses[1] == F(117, 50)
    assert trajectory(eta=F(1, 4), iterations=1)[1] == [4, F(9, 8)]
    boundary, boundary_losses = trajectory(eta=F(1, 2))
    assert boundary[1:3] == [(F(1), F(-1)), (F(1, 2), F(1))]
    assert boundary_losses[:3] == [4, F(5, 2), F(17, 8)]
    assert all(loss > 2 for loss in boundary_losses)
    zero_mode, zero_mode_losses = trajectory((2, 0), F(3, 4))
    assert zero_mode[1] == (F(1, 2), F(0))
    assert all(b < a for a, b in zip(zero_mode_losses, zero_mode_losses[1:]))
    divergent, _ = trajectory((2, 1), F(3, 4), 4)
    assert divergent[-1][1] == 16
    alternating, alternating_losses = trajectory(eta=F(3, 8))
    assert alternating[1] == (F(5, 4), F(-1, 2))
    assert all(b < a for a, b in zip(alternating_losses, alternating_losses[1:]))
    balanced, balanced_losses = trajectory(eta=F(2, 5), iterations=2)
    assert balanced[1] == (F(6, 5), F(-3, 5))
    assert balanced_losses == [4, F(36, 25), F(324, 625)]
    assert factors(F(1, 8), a=8) == (0, F(1, 2))
    assert trajectory((2, 1), F(1, 8), 1, a=8, c=1)[0][1] == (1, F(1, 2))
    assert F(2, max(8, 4)) == F(1, 4)
    for invalid in (0, -1):
        try:
            factors(invalid)
        except ValueError:
            pass
        else:
            raise AssertionError("Nonpositive learning rate accepted")
    print("Exact default factors at eta=1/4:", factors(F(1, 4)))
    print("Exact eta=1/2 boundary path:", boundary[:4], "; y does not decay")
    print("All-start stability: 0 < eta < 2/max(a,4), for a>0.")
    print("A zero starting component can hide an unstable mode; finite runs are not proofs.")


if __name__ == "__main__":
    main()

"""Vectorized binary64 GD paths for many rates, checked by a closed-form power grid."""
import numpy as np


def trajectories(start, rates, iterations=12, a=1.0, c=0.0):
    start, rates = np.asarray(start, dtype=np.float64), np.asarray(rates, dtype=np.float64)
    if start.shape != (2,) or rates.ndim != 1 or not rates.size:
        raise ValueError("Require one two-component start and a nonempty rate vector")
    if not np.all(np.isfinite(start)) or not np.all(np.isfinite(rates)) or np.any(rates <= 0):
        raise ValueError("Require finite start and positive finite rates")
    if not np.isfinite(a) or a <= 0 or not np.isfinite(c):
        raise ValueError("Require positive finite a and finite c")
    if type(iterations) is not int or not 0 <= iterations <= 1000:
        raise ValueError("Require an integer iteration count in [0,1000]")
    center, hessian = np.array([c, 0.0]), np.diag([a, 4.0])
    path = np.empty((iterations+1, rates.size, 2), dtype=np.float64)
    path[0] = start
    for k in range(iterations):
        path[k+1] = path[k] - rates[:, None]*((path[k]-center) @ hessian)
        if not np.all(np.isfinite(path[k+1])):
            raise OverflowError(f"Non-finite binary64 iterate at step {k+1}")
    shifted = path-center
    losses = np.einsum("kni,ij,knj->kn", shifted, hessian, shifted)/2
    if not np.all(np.isfinite(losses)):
        raise OverflowError("Non-finite binary64 loss")
    return path, losses


def main():
    rates = np.array([0.03, 0.25, 0.4, 0.5, 0.75])
    path, losses = trajectories([2, 1], rates)
    factors = 1-rates[:, None]*np.array([1.0, 4.0])
    powers = np.arange(13)[:, None, None]
    direct = np.array([2.0, 1.0])*factors[None, :, :]**powers
    np.testing.assert_allclose(path, direct, rtol=2e-14, atol=1e-14)
    np.testing.assert_array_equal(path[1, 1], [1.5, 0])
    np.testing.assert_array_equal(path[:3, 3, 1], [1, -1, 1])
    assert np.all(np.diff(losses[:, :3], axis=0) < 0)
    assert path[-1, 4, 1] == 4096
    exception, _ = trajectories([2, 0], [0.75], 8)
    np.testing.assert_array_equal(exception[:, 0, 1], np.zeros(9))
    assert exception[-1, 0, 0] == 2*(0.25**8)
    np.testing.assert_array_equal(np.linalg.eigvalsh(np.diag([1.0, 4.0])), [1, 4])
    for invalid in ([0], [-1], [np.nan]):
        try:
            trajectories([2, 1], invalid)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid rate accepted")
    print("Binary64 final losses by rate ≈", losses[-1])
    print("The eta=0.5 y mode alternates without decay; eta=0.75 grows unless y starts at zero.")
    print("Closed-form cross-check: rtol=2e-14, atol=1e-14 on this bounded grid.")


if __name__ == "__main__":
    main()

Explore: predict an active mode before running

Compare the slow preset with the canonical run. Identify the error factor closest to magnitude 1, then predict which coordinate will dominate after many steps. On the default quadratic the x curvature is smaller, so a small rate can erase y error sooner while leaving x progress slow.

Choose damped oscillation and count sign changes in y while comparing its magnitude. Then choose no decay and repeat the comparison. Finally choose the divergent preset with y set to zero, predict the exceptional behavior, and introduce a small nonzero y start to excite the unstable mode.

Move the center c while leaving a and the rate fixed. The x error remains $x-c$; the multiplier does not change. As a hand extension, use $a=8,c=1$, rate $1/8$, and start $(2,1)$. The x error disappears immediately and y halves, producing $(1,1/2)$. The all-start upper boundary is now $1/4$, set by the x curvature rather than y.

Common mistakes and counterexamples

The rate is not the error multiplier. Different curvatures convert the same rate into different factors. Also apply the x factor to $x-c$, not to x, when the center is shifted.

Oscillation does not imply divergence: a factor of $-1/2$ contracts, while $-1$ does not decay and $-2$ grows. A short decreasing loss trace does not imply parameter convergence, as the $\eta=1/2$ example demonstrates. A zero starting mode can also make an unstable rate appear safe until that mode is perturbed.

Numerical considerations

The convergence derivation concerns exact real arithmetic. A binary64 trajectory can stagnate when a displacement rounds away, underflow to zero, overflow, or stop at a finite iteration budget. These are computational events, distinct from the behavior of $r^k$ as $k\to\infty$.

Near a boundary, a small rate or curvature perturbation can move a factor from slightly inside to slightly outside magnitude 1. Compare the unrounded state and a stated numerical error scale rather than a rounded displayed factor. An exactly absent mode is also a special case; small representation or input errors in that coordinate can matter greatly when its factor has magnitude above 1.

The references label numerical trajectories binary64 and reject non-finite states. Their finite tolerance checks verify implementation on a bounded set of runs; the tolerance is not part of the mathematical definition of stability.

Engineering and ML connections

The Hessian describes directional curvature, and gradient descent turns that curvature into repeated errors. For a positive-definite quadratic, a large ratio of largest to smallest curvature can force a shared stable rate that contracts the gentle direction slowly. Parameter scaling can therefore change how efficiently the same optimizer moves.

A neural-network objective usually lacks one constant Hessian valid across the full path, so this quadratic interval must not be presented as its global safety certificate. It is a local mechanism to inspect: changes in curvature, gradient scale, and parameter units can change a rate’s behavior. The next chapter adds momentum, whose extra memory changes the recurrence and requires its own analysis.

Exercises

Calculate factors before predicting a path. State whether a claim concerns every start, an active mode, a finite numerical run, or an exact limit.

Summary and checkpoints

A quadratic gradient step multiplies each error mode by $1-\eta\lambda$. Check that you can derive the all-start interval, distinguish shrinking oscillation from nondecay, recover the boundary’s limiting loss $2$, and explain why an absent unstable mode is an exception. A rate controls a complete trajectory, and its first few favorable updates do not establish the limiting outcome.