Numerical stopping and optimization diagnostics

Inspect why this numerical run stopped

Parameter space: θ₁ and θ₂

Loss slice: θ₂ = 0

Current parameter vector: (2, 1)
Current objective value: 4
Actual gradient components: (2, 4)
Learning rate: 0.250000
Update vector: (−0.5, −1)
Next parameter vector: (1.5, 0)
New objective value: 1.125000
Selected raw gradient norm: 4.472136
Final raw gradient norm at stop: 0.084470
Raw gradient stopping threshold: 0.1
Absolute loss-change stopping threshold: disabled (0)
Last accepted absolute loss change: 0.002775
Explicit divergence norm boundary: 1000000
Actual termination reason: gradient-norm threshold (not a global-optimality certificate)

≈ binary64 calculation · = analytic classification. Replay selects the same evaluated step in every view.

Iteration 0: at θ ≈ (2, 1), L ≈ 4 and gradient ≈ (2, 4). Rate η ≈ 0.25 gives update ≈ (−0.5, −1). Adding that update gives θ next ≈ (1.5, 0) and L next ≈ 1.125. The step follows the negative gradient; a finite step can still increase loss. Stop: gradient-norm threshold (not a global-optimality certificate). Parameters outside the displayed window are retained in the table; the view never changes the path. Raw gradient threshold = 0.1; absolute loss-change threshold = 0; clip norm = 0 (0 disables each). Parameter norm boundary = 1000000. Numerical stopping and clipping do not certify a global minimum.

L=\tfrac a2(\theta_1-c)^2+2\theta_2^2

\theta_{k+1}=\theta_k-\eta\nabla L(\theta_k)

\theta_{1}\approx \begin{bmatrix}2\\1\end{bmatrix}+\begin{bmatrix}-0.5\\-1\end{bmatrix}\approx \begin{bmatrix}1.5\\0\end{bmatrix}

\|\nabla L(\theta_k)\|\le\tau_g,\quad|L_{k+1}-L_k|\le\tau_L

Evaluated binary64 history. Replay changes the selected row, never the stored mathematics.
k	θ(k)	L(k)	gradient	η	update	θ(k+1)	L(k+1)	raw gradient norm	clip scale	applied gradient	unclipped update	absolute loss change
0	(2, 1)	4	(2, 4)	0.25	(−0.5, −1)	(1.5, 0)	1.125	4.472136	1	(2, 4)	(−0.5, −1)	2.875
1	(1.5, 0)	1.125	(1.5, 0)	0.25	(−0.375, 0)	(1.125, 0)	0.632813	1.5	1	(1.5, 0)	(−0.375, 0)	0.492188
2	(1.125, 0)	0.632813	(1.125, 0)	0.25	(−0.28125, 0)	(0.84375, 0)	0.355957	1.125	1	(1.125, 0)	(−0.28125, 0)	0.276855
3	(0.84375, 0)	0.355957	(0.84375, 0)	0.25	(−0.210938, 0)	(0.632813, 0)	0.200226	0.84375	1	(0.84375, 0)	(−0.210938, 0)	0.155731
4	(0.632813, 0)	0.200226	(0.632813, 0)	0.25	(−0.158203, 0)	(0.474609, 0)	0.112627	0.632813	1	(0.632813, 0)	(−0.158203, 0)	0.087599
5	(0.474609, 0)	0.112627	(0.474609, 0)	0.25	(−0.118652, 0)	(0.355957, 0)	0.063353	0.474609	1	(0.474609, 0)	(−0.118652, 0)	0.049274
6	(0.355957, 0)	0.063353	(0.355957, 0)	0.25	(−0.088989, 0)	(0.266968, 0)	0.035636	0.355957	1	(0.355957, 0)	(−0.088989, 0)	0.027717
7	(0.266968, 0)	0.035636	(0.266968, 0)	0.25	(−0.066742, 0)	(0.200226, 0)	0.020045	0.266968	1	(0.266968, 0)	(−0.066742, 0)	0.015591
8	(0.200226, 0)	0.020045	(0.200226, 0)	0.25	(−0.050056, 0)	(0.150169, 0)	0.011275	0.200226	1	(0.200226, 0)	(−0.050056, 0)	0.00877
9	(0.150169, 0)	0.011275	(0.150169, 0)	0.25	(−0.037542, 0)	(0.112627, 0)	0.006342	0.150169	1	(0.150169, 0)	(−0.037542, 0)	0.004933
10	(0.112627, 0)	0.006342	(0.112627, 0)	0.25	(−0.028157, 0)	(0.08447, 0)	0.003568	0.112627	1	(0.112627, 0)	(−0.028157, 0)	0.002775

Evaluated iteration 0

\theta_{1}\approx \begin{bmatrix}2\\1\end{bmatrix}+\begin{bmatrix}-0.5\\-1\end{bmatrix}\approx \begin{bmatrix}1.5\\0\end{bmatrix}

SpeedStep 1 of 11

Parameter θ₁

from −3 to 3, up to 2 decimal places

Parameter θ₂

from −3 to 3, up to 2 decimal places

Quadratic curvature a

from 0.25 to 4, up to 2 decimal places

Quadratic center c

from −2 to 2, up to 2 decimal places

Learning rate η

from 0.01 to 2, up to 2 decimal places

Iteration budget

from 1 to 40, whole numbers only

Raw gradient tolerance (0 disables)

from 0 to 1, up to 2 decimal places

Absolute loss-change tolerance (0 disables)

from 0 to 1, up to 2 decimal places

Gradient clip norm (0 disables)

from 0 to 5, up to 2 decimal places

Inspect the raw gradient and its norm before an update, then compare unclipped and applied displacements. Replay finite loss changes and read the precise stopping reason with its configured thresholds.

Reset starts at $(2,1)$ on $L=x^2/2+2y^2$ , with $\eta=1/4$ , $K=12$ , raw-gradient tolerance $0.1$ , loss-change tolerance 0, clipping threshold 0, and divergence norm bound $10^6$ . Predict whether the first move or the first gradient check ends the run. Then compare the small-loss preset, which stops after two finite updates while its gradient is still far from zero.

Intuition: a diagnostic is evidence about a computation

An optimizer may stop because its measured slope is small, its objective changes little, its work budget is used, or its arithmetic becomes unsafe. These are distinct observations. The reason should name the event and show the values used to make the decision.

A small gradient describes local first-order behavior. A tiny loss change may result from a tiny learning rate, rounding, or motion along an almost flat direction. A finite budget says how much work was allowed. None of those observations, by itself, proves a global optimum.

Gradient clipping changes a numerical update while retaining the raw gradient for diagnosis. It can limit one step’s magnitude, but it does not turn an invalid objective into a valid one or establish that repeated steps converge.

Definition: two optional thresholds and one update intervention

This chapter uses plain gradient descent, with no momentum, Adam history, stochastic sampling, projection, or penalty. For the quadratic family $L=a(x-c)^2/2+2y^2$ , the raw gradient is $g=(a(x-c),4y)$ .

A positive raw-gradient tolerance $\tau_g$ stops before an update when

\norm{g_k}_2\leq\tau_g.

A positive absolute loss-change tolerance $\tau_L$ stops after a finite update when

\abs{L(\theta_{k+1})-L(\theta_k)}\leq\tau_L.

Zero disables the corresponding tolerance test. It does not turn exact zero into an enabled threshold or disable finite-value checks. An exactly zero raw gradient has its own explicit stop when the gradient-tolerance test is disabled.

For clipping threshold $C>0$ and nonzero raw g, define

s=\min\left(1,\frac C{\norm g_2}\right),\qquad g_{\mathrm{applied}}=sg,\qquad \Delta_{\mathrm{unclipped}}=-\eta g,\qquad \Delta_{\mathrm{applied}}=-\eta sg.

At $C=0$ , clipping is disabled and $s=1$ . A zero raw gradient stops before the quotient is formed. The positive divergence bound B triggers when the next finite point has $\norm{\theta_{k+1}}_2>B$ .

Mechanics: the policy order is part of the meaning

At each permitted plain-descent iteration, apply these checks in order:

Evaluate the current objective on its domain and its raw gradient. Stop if the objective or required gradient is undefined or non-finite.
If $\tau_g>0$ , compare the raw gradient norm with $\tau_g$ before any update.
If the raw gradient is exactly zero, stop explicitly. This reason carries no automatic optimum claim.
Form the applied gradient, displacement, and next point. Guard the actual transition for finite values.
Evaluate the next objective and required gradient. A non-finite next evaluation stops before a loss difference is used.
Check the next finite point against the configured divergence norm bound.
If $\tau_L>0$ , compare the finite endpoint losses using their absolute difference.
If K updates have been completed, stop for the iteration budget.

The first applicable reason wins. A positive gradient tolerance also catches exact zero before the separate zero-gradient rule. Divergence takes precedence over a small loss change on the same finite transition. Loss-change stopping takes precedence over budget exhaustion on the last allowed update.

Keep raw g, its norm, the clipping scale, applied g, and both displacement vectors in the record. The raw-gradient test must not use the capped applied norm: otherwise lowering the clipping threshold could manufacture a misleading tolerance stop.

Replay selects these recorded evaluations and interventions. A failed candidate remains an attempted transition with its failure reason; it must not be presented as a successful finite loss evaluation.

Derivation: interpret each measured quantity separately

For the default quadratic and rate $1/4$ , the error factors are $(3/4,0)$ . After the first update,

x_k=2(3/4)^k,\qquad y_k=0,\qquad L_k=2(3/4)^{2k},\qquad k\geq1.

The gradient norm is then $\abs{x_k}$ . The loss changes with a different scale:

L_k-L_{k+1}=\left(1-\frac9{16}\right)L_k =\frac7{16}L_k.

Thus a loss-change threshold and gradient threshold generally stop at different states. Rescaling the objective scales both gradients and loss differences; changing parameter units also changes Euclidean gradient norms. Tolerances need a stated scale and purpose.

Clipping preserves the direction of a nonzero raw gradient and yields

\norm{g_{\mathrm{applied}}}_2=\min(\norm g_2,C),\qquad \norm{\Delta_{\mathrm{applied}}}_2 =\eta\min(\norm g_2,C).

It limits displacement at that iteration. It is not a coordinate clamp, a projection onto a parameter region, or a change to the objective. A very large finite gradient can still reveal trouble even when its applied update is small.

The absolute loss-change rule does not require a decrease. A small increase also satisfies the tolerance. A divergence bound is likewise a configured stop for excessive parameter norm, not a proof that the exact sequence tends to infinity.

Worked examples: one path, different stopping reasons

The canonical first gradient and update are

g_0=(2,4),\quad \norm{g_0}_2=\sqrt{20}>0.1,\quad \Delta_0=(-1/2,-1),\quad \theta_1=(3/2,0),\quad L_1=9/8.

The next point is $(9/8,0)$ with loss $81/128$ . For the canonical raw-gradient tolerance, after ten updates the norm is $2(3/4)^{10}\approx0.112627>0.1$ . After eleven it is

2(3/4)^{11}=\frac{177147}{2097152}\approx0.0844703.

The next pre-update check stops at that state. Eleven updates were completed; a twelfth update is not taken. The positive final loss means the exact zero-loss minimizer has not been reached.

Worked example: Inspect why this numerical run stopped

Parameter space: θ₁ and θ₂

Loss slice: θ₂ = 0

Current parameter vector: (2, 1)
Current objective value: 4
Actual gradient components: (2, 4)
Learning rate: 0.250000
Update vector: (−0.5, −1)
Next parameter vector: (1.5, 0)
New objective value: 1.125000
Selected raw gradient norm: 4.472136
Final raw gradient norm at stop: 0.084470
Raw gradient stopping threshold: 0.1
Absolute loss-change stopping threshold: disabled (0)
Last accepted absolute loss change: 0.002775
Explicit divergence norm boundary: 1000000
Actual termination reason: gradient-norm threshold (not a global-optimality certificate)

≈ binary64 calculation · = analytic classification. Replay selects the same evaluated step in every view.

L=\tfrac a2(\theta_1-c)^2+2\theta_2^2

\theta_{k+1}=\theta_k-\eta\nabla L(\theta_k)

\theta_{1}\approx \begin{bmatrix}2\\1\end{bmatrix}+\begin{bmatrix}-0.5\\-1\end{bmatrix}\approx \begin{bmatrix}1.5\\0\end{bmatrix}

\|\nabla L(\theta_k)\|\le\tau_g,\quad|L_{k+1}-L_k|\le\tau_L

Evaluated binary64 history. Replay changes the selected row, never the stored mathematics.
k	θ(k)	L(k)	gradient	η	update	θ(k+1)	L(k+1)	raw gradient norm	clip scale	applied gradient	unclipped update	absolute loss change
0	(2, 1)	4	(2, 4)	0.25	(−0.5, −1)	(1.5, 0)	1.125	4.472136	1	(2, 4)	(−0.5, −1)	2.875
1	(1.5, 0)	1.125	(1.5, 0)	0.25	(−0.375, 0)	(1.125, 0)	0.632813	1.5	1	(1.5, 0)	(−0.375, 0)	0.492188
2	(1.125, 0)	0.632813	(1.125, 0)	0.25	(−0.28125, 0)	(0.84375, 0)	0.355957	1.125	1	(1.125, 0)	(−0.28125, 0)	0.276855
3	(0.84375, 0)	0.355957	(0.84375, 0)	0.25	(−0.210938, 0)	(0.632813, 0)	0.200226	0.84375	1	(0.84375, 0)	(−0.210938, 0)	0.155731
4	(0.632813, 0)	0.200226	(0.632813, 0)	0.25	(−0.158203, 0)	(0.474609, 0)	0.112627	0.632813	1	(0.632813, 0)	(−0.158203, 0)	0.087599
5	(0.474609, 0)	0.112627	(0.474609, 0)	0.25	(−0.118652, 0)	(0.355957, 0)	0.063353	0.474609	1	(0.474609, 0)	(−0.118652, 0)	0.049274
6	(0.355957, 0)	0.063353	(0.355957, 0)	0.25	(−0.088989, 0)	(0.266968, 0)	0.035636	0.355957	1	(0.355957, 0)	(−0.088989, 0)	0.027717
7	(0.266968, 0)	0.035636	(0.266968, 0)	0.25	(−0.066742, 0)	(0.200226, 0)	0.020045	0.266968	1	(0.266968, 0)	(−0.066742, 0)	0.015591
8	(0.200226, 0)	0.020045	(0.200226, 0)	0.25	(−0.050056, 0)	(0.150169, 0)	0.011275	0.200226	1	(0.200226, 0)	(−0.050056, 0)	0.00877
9	(0.150169, 0)	0.011275	(0.150169, 0)	0.25	(−0.037542, 0)	(0.112627, 0)	0.006342	0.150169	1	(0.150169, 0)	(−0.037542, 0)	0.004933
10	(0.112627, 0)	0.006342	(0.112627, 0)	0.25	(−0.028157, 0)	(0.08447, 0)	0.003568	0.112627	1	(0.112627, 0)	(−0.028157, 0)	0.002775

Evaluated iteration 0

\theta_{1}\approx \begin{bmatrix}2\\1\end{bmatrix}+\begin{bmatrix}-0.5\\-1\end{bmatrix}\approx \begin{bmatrix}1.5\\0\end{bmatrix}

SpeedStep 1 of 11

Replay canonical plain descent: (2,1) to (3/2,0) to (9/8,0). The positive raw-gradient tolerance ends the run before a twelfth update, after eleven completed updates.

The small-loss preset disables gradient stopping and sets $\tau_L=1/2$ . The first loss change is $4-9/8=23/8$ , above the threshold. The second is

9/8-81/128=63/128<1/2.

It stops after two updates at $(9/8,0)$ , where the raw gradient norm is still $9/8$ . The budget preset uses the same path with both tolerances disabled and $K=2$ ; it ends at the same point for the different reason of iteration budget.

For the clipping preset, retain start $(2,1)$ and rate $1/4$ , disable gradient stopping, and set $C=1$ . At the first step,

s=\frac1{\sqrt{20}},\qquad g_{\mathrm{applied}}=\frac{(2,4)}{\sqrt{20}},\qquad \Delta_{\mathrm{applied}}=\frac{(-1/2,-1)}{\sqrt{20}}.

The next point is $(2-1/(2\sqrt{20}),1-1/\sqrt{20})$ , and the displacement norm is $1/4$ . The unclipped vector remains $(-1/2,-1)$ in the record, and the raw norm remains $\sqrt{20}$ .

The divergent preset uses the quadratic, $\eta=2$ , $K=12$ , both tolerances 0, no clipping, and bound $10^6$ . Its factors are $(-1,-7)$ : x alternates between $\pm2$ , while $\abs{y_k}=7^k$ . The eighth next point is $(2,5764801)$ , exceeding the bound, so the run stops after eight finite updates.

The non-finite preset instead uses $L=x^4+y^4$ , start $(3,1)$ , $\eta=2$ , $K=12$ , both tolerances 0, no clipping, and the explicitly configured bound $10^{300}$ . Its recurrence is $x_{k+1}=x_k-8x_k^3$ , likewise for y. The first two next points are $(-213,-7)$ and $(77308563,2737)$ . The fifth proposed point is finite in binary64, but its quartic objective overflows. The next-evaluation guard wins before divergence or a loss comparison; the failed loss is not replaced by zero.

Executable reference code

The standalone standard-library program records the actual binary64 plain-descent recurrence and its ordered reasons. Independent Fraction calculations check the canonical path, raw-gradient stopping count, two-step loss-change values, and budget endpoint. An 80-digit Decimal calculation checks the clipped first step independently.

Literal assertions also cover an explicit zero gradient at a saddle, loss-change precedence on the last budgeted step, divergence before loss-change stopping, a non-finite transition, and a non-finite next objective with finite candidate coordinates. The NumPy reference computes actual two-component array gradients and updates, then compares valid recorded values with a separate high-precision scalar recurrence. Its checked binary64 tolerance is relative and absolute $10^{-12}$ ; stop reasons and counts must match exactly.

code/python/optimization_diagnostics.py

"""Plain binary64 descent with ordered stops; independent Fraction and Decimal checks."""
from decimal import Decimal as D, localcontext
from fractions import Fraction as F
from math import hypot, isfinite


def evaluate(point, objective, a, c):
    x, y = point
    if objective == "quadratic":
        loss, gradient = a*(x-c)*(x-c)/2+2*y*y, (a*(x-c), 4*y)
    elif objective == "flat":
        loss, gradient = x*x*x*x+y*y*y*y, (4*x*x*x, 4*y*y*y)
    elif objective == "saddle":
        loss, gradient = x*x-y*y, (2*x, -2*y)
    else:
        raise ValueError("Objective must be quadratic, flat, or saddle")
    return (loss, gradient) if all(isfinite(value) for value in (loss, *gradient)) else None


def run(start=(2, 1), eta=0.25, iterations=12, a=1.0, c=0.0, objective="quadratic",
        gradient_tol=0.1, loss_tol=0.0, clip=0.0, norm_bound=1e6):
    if len(start) != 2 or type(iterations) is not int or not 1 <= iterations <= 40:
        raise ValueError("Require two coordinates and an integer budget in [1,40]")
    requested = point = tuple(map(float, start))
    eta, a, c, gradient_tol, loss_tol, clip, norm_bound = map(
        float, (eta, a, c, gradient_tol, loss_tol, clip, norm_bound))
    if not all(isfinite(value) for value in (*point, eta, a, c, gradient_tol, loss_tol, clip, norm_bound)):
        raise ValueError("Require finite inputs")
    if eta <= 0 or not 0.25 <= a <= 4 or not -2 <= c <= 2 or min(gradient_tol, loss_tol, clip) < 0 or norm_bound <= 0:
        raise ValueError("Require eta,bound>0, a in [0.25,4], c in [-2,2], and nonnegative thresholds")
    if objective not in ("quadratic", "flat", "saddle"):
        raise ValueError("Objective must be quadratic, flat, or saddle")
    frames, reason = [], "iteration budget"
    for k in range(iterations):
        current = evaluate(point, objective, a, c)
        if current is None:
            reason = "non-finite current evaluation"
            break
        loss, raw = current
        norm = hypot(*raw)
        if not isfinite(norm):
            reason = "non-finite current evaluation"
            break
        if gradient_tol > 0 and norm <= gradient_tol:
            reason = "gradient tolerance"
            break
        if norm == 0:
            reason = "zero gradient"
            break
        scale = min(1.0, clip/norm) if clip > 0 else 1.0
        applied = (scale*raw[0], scale*raw[1])
        raw_update = (-eta*raw[0], -eta*raw[1])
        raw_proposal = (point[0]+raw_update[0], point[1]+raw_update[1])
        update = (-eta*applied[0], -eta*applied[1])
        next_point = (point[0]+update[0], point[1]+update[1])
        frame = {"k": k, "point": point, "loss": loss, "raw_gradient": raw, "gradient_norm": norm,
                 "scale": scale, "applied_gradient": applied, "raw_update": raw_update,
                 "raw_proposal": raw_proposal, "applied_update": update, "next": next_point,
                 "next_loss": None, "next_gradient": None}
        frames.append(frame)
        if not all(isfinite(value) for value in (scale, *applied, *update, *next_point)):
            reason = "non-finite transition"
            break
        next_evaluation = evaluate(next_point, objective, a, c)
        if next_evaluation is None:
            reason = "non-finite next evaluation"
            break
        next_loss, next_gradient = next_evaluation
        frame["next_loss"], frame["next_gradient"] = next_loss, next_gradient
        point = next_point
        if hypot(*point) > norm_bound:
            reason = "divergence bound"
            break
        if loss_tol > 0 and abs(next_loss-loss) <= loss_tol:
            reason = "loss change tolerance"
            break
    return {"start": requested, "point": point, "frames": frames, "reason": reason,
            "final_evaluation": evaluate(point, objective, a, c)}


def main():
    canonical = run()
    assert canonical["reason"] == "gradient tolerance" and len(canonical["frames"]) == 11
    assert canonical["point"] == (float(F(177147, 2097152)), 0)
    assert 2*F(3, 4)**10 > F(1, 10) and 2*F(3, 4)**11 < F(1, 10)
    for k, frame in enumerate(canonical["frames"]):
        current = (F(2), F(1)) if k == 0 else (2*F(3, 4)**k, F(0))
        next_point = (2*F(3, 4)**(k+1), F(0))
        assert frame["point"] == tuple(map(float, current))
        assert frame["next"] == tuple(map(float, next_point))
        assert frame["loss"] == float(current[0]**2/2+2*current[1]**2)
        assert frame["next_loss"] == float(next_point[0]**2/2)
        assert frame["raw_update"] == frame["applied_update"]
        assert frame["raw_proposal"] == frame["next"]
    early = run(start=(0.05, 0))
    assert early["reason"] == "gradient tolerance" and not early["frames"]
    assert early["point"] == (0.05, 0)
    inclusive = run(start=(1.5, 0), gradient_tol=1.5)
    assert inclusive["reason"] == "gradient tolerance" and not inclusive["frames"]
    small_loss = run(gradient_tol=0, loss_tol=0.5)
    assert small_loss["reason"] == "loss change tolerance" and len(small_loss["frames"]) == 2
    assert small_loss["point"] == (1.125, 0)
    assert small_loss["frames"][1]["next_loss"] == float(F(81, 128))
    assert abs(small_loss["frames"][1]["next_loss"]-small_loss["frames"][1]["loss"]) == float(F(63, 128))
    budget = run(iterations=2, gradient_tol=0)
    assert budget["reason"] == "iteration budget" and budget["point"] == (1.125, 0)
    assert len(budget["frames"]) == 2
    assert run(iterations=2, gradient_tol=0, loss_tol=0.5)["reason"] == "loss change tolerance"

    clipped = run(gradient_tol=0, clip=1)
    assert clipped["reason"] == "iteration budget" and len(clipped["frames"]) == 12
    first = clipped["frames"][0]
    assert first["raw_gradient"] == (2, 4) and first["gradient_norm"] == hypot(2, 4)
    assert first["raw_update"] == (-0.5, -1) and first["raw_proposal"] == (1.5, 0)
    with localcontext() as context:
        context.prec = 80
        scale = 1/D(20).sqrt()
        for actual, expected in zip(first["applied_gradient"], (2*scale, 4*scale)):
            assert abs(actual-float(expected)) < 1e-15
        for actual, expected in zip(first["next"], (2-D("0.5")*scale, 1-scale)):
            assert abs(actual-float(expected)) < 1e-15
    assert abs(hypot(*first["applied_update"])-0.25) < 1e-15
    assert all(frame["gradient_norm"] > 0 and hypot(*frame["applied_gradient"]) <= 1+1e-15 for frame in clipped["frames"])
    assert run(gradient_tol=1.1, clip=1)["frames"]  # The capped norm must not trigger a raw-gradient stop.
    hand = run(start=(3, 1), gradient_tol=0, clip=2, iterations=1)["frames"][0]
    assert hand["gradient_norm"] == 5 and hand["scale"] == 0.4
    assert abs(hand["next"][0]-2.7) < 1e-15 and abs(hand["next"][1]-0.6) < 1e-15

    divergent = run(eta=2, gradient_tol=0)
    assert divergent["reason"] == "divergence bound" and len(divergent["frames"]) == 8
    assert divergent["point"] == (2, 5764801)
    precedence = run(eta=2, gradient_tol=0, loss_tol=1e100, norm_bound=1)
    assert precedence["reason"] == "divergence bound" and len(precedence["frames"]) == 1
    saddle = run(start=(0, 0), objective="saddle", gradient_tol=0)
    assert saddle["reason"] == "zero gradient" and not saddle["frames"]
    assert evaluate((0, 1), "saddle", 1, 0)[0] == -1
    assert run(start=(0, 0))["reason"] == "gradient tolerance"
    nonfinite = run(start=(3, 1), objective="flat", eta=2, gradient_tol=0, norm_bound=1e300)
    assert nonfinite["reason"] == "non-finite next evaluation" and len(nonfinite["frames"]) == 5
    assert nonfinite["frames"][0]["next"] == (-213, -7)
    assert nonfinite["frames"][1]["next"] == (77308563, 2737)
    failed = nonfinite["frames"][-1]
    assert all(isfinite(value) for value in failed["next"])
    assert hypot(*failed["next"]) < 1e300 and failed["next_loss"] is None
    assert nonfinite["point"] == failed["point"]  # Preserve the last valid point, alongside the failed candidate.
    assert run(eta=1e308, gradient_tol=0)["reason"] == "non-finite transition"
    invalid_current = run(start=(1e100, 0), objective="flat", gradient_tol=1e308)
    assert invalid_current["reason"] == "non-finite current evaluation" and not invalid_current["frames"]

    for settings in ({"gradient_tol": -1}, {"clip": -1}, {"loss_tol": -1}, {"norm_bound": 0},
                     {"iterations": 41}, {"eta": 0}, {"start": (float("nan"), 0)}, {"objective": "unknown"}):
        try:
            run(**settings)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid input accepted")
    print("Canonical gradient stop: 11 updates; small-loss stop: 2; budget stop: 2.")
    print("Clipping: raw norm sqrt(20), scale 1/sqrt(20), applied displacement norm 1/4.")
    print("Divergence: 8 finite updates; quartic: 5 attempts, last finite candidate has non-finite next evaluation.")
    print("Saddle: explicit zero-gradient stop after 0 updates; no optimum claim. Ordered precedence checks passed.")


if __name__ == "__main__":
    main()

code/python/optimization_diagnostics_numpy.py

"""Actual NumPy plain descent and ordered stops, cross-checked with 80-digit scalar arithmetic."""
from decimal import Decimal as D, localcontext
from fractions import Fraction as F
import numpy as np


def evaluate(point, objective, a, c):
    x, y = point
    if objective == "quadratic":
        loss, gradient = a*(x-c)*(x-c)/2+2*y*y, np.array([a*(x-c), 4*y])
    elif objective == "flat":
        loss, gradient = np.sum(point*point*point*point), 4*point*point*point
    elif objective == "saddle":
        loss, gradient = x*x-y*y, np.array([2*x, -2*y])
    else:
        raise ValueError("Objective must be quadratic, flat, or saddle")
    return (float(loss), gradient) if np.all(np.isfinite([loss, *gradient])) else None


def run(start=(2, 1), eta=0.25, iterations=12, a=1.0, c=0.0, objective="quadratic",
        gradient_tol=0.1, loss_tol=0.0, clip=0.0, norm_bound=1e6):
    requested = point = np.asarray(start, dtype=np.float64).copy()
    if point.shape != (2,) or not np.all(np.isfinite(point)):
        raise ValueError("Require a finite two-coordinate start")
    if type(iterations) is not int or not 1 <= iterations <= 40:
        raise ValueError("Require an integer budget in [1,40]")
    eta, a, c, gradient_tol, loss_tol, clip, norm_bound = map(
        float, (eta, a, c, gradient_tol, loss_tol, clip, norm_bound))
    if not np.all(np.isfinite([eta, a, c, gradient_tol, loss_tol, clip, norm_bound])):
        raise ValueError("Require finite parameters")
    if eta <= 0 or not 0.25 <= a <= 4 or not -2 <= c <= 2 or min(gradient_tol, loss_tol, clip) < 0 or norm_bound <= 0:
        raise ValueError("Require eta,bound>0, a in [0.25,4], c in [-2,2], and nonnegative thresholds")
    if objective not in ("quadratic", "flat", "saddle"):
        raise ValueError("Objective must be quadratic, flat, or saddle")
    frames, reason = [], "iteration budget"
    with np.errstate(over="ignore", invalid="ignore"):
        for k in range(iterations):
            current = evaluate(point, objective, a, c)
            if current is None:
                reason = "non-finite current evaluation"
                break
            loss, raw = current
            norm = float(np.hypot(*raw))
            if not np.isfinite(norm):
                reason = "non-finite current evaluation"
                break
            if gradient_tol > 0 and norm <= gradient_tol:
                reason = "gradient tolerance"
                break
            if norm == 0:
                reason = "zero gradient"
                break
            scale = min(1.0, clip/norm) if clip > 0 else 1.0
            applied = scale*raw
            raw_update = -eta*raw
            raw_proposal = point+raw_update
            update = -eta*applied
            next_point = point+update
            frame = {"k": k, "point": point, "loss": loss, "raw_gradient": raw,
                     "gradient_norm": norm, "scale": scale, "applied_gradient": applied,
                     "raw_update": raw_update, "raw_proposal": raw_proposal, "applied_update": update,
                     "next": next_point, "next_loss": None, "next_gradient": None}
            frames.append(frame)
            if not np.all(np.isfinite([scale, *applied, *update, *next_point])):
                reason = "non-finite transition"
                break
            next_evaluation = evaluate(next_point, objective, a, c)
            if next_evaluation is None:
                reason = "non-finite next evaluation"
                break
            next_loss, next_gradient = next_evaluation
            frame["next_loss"], frame["next_gradient"] = next_loss, next_gradient
            point = next_point
            if np.hypot(*point) > norm_bound:
                reason = "divergence bound"
                break
            if loss_tol > 0 and abs(next_loss-loss) <= loss_tol:
                reason = "loss change tolerance"
                break
        final = evaluate(point, objective, a, c)
    return {"start": requested, "point": point, "frames": frames, "reason": reason, "final_evaluation": final}


def main():
    def check(actual, expected):
        np.testing.assert_allclose(actual, np.asarray(expected, dtype=np.float64), rtol=1e-12, atol=1e-12)

    cases = [
        ({}, "gradient tolerance", 11),
        ({"gradient_tol": 0, "loss_tol": 0.5}, "loss change tolerance", 2),
        ({"gradient_tol": 0, "iterations": 2}, "iteration budget", 2),
        ({"gradient_tol": 0, "clip": 1}, "iteration budget", 12),
        ({"gradient_tol": 0, "eta": 2}, "divergence bound", 8),
        ({"start": (3, 1), "objective": "flat", "eta": 2, "gradient_tol": 0, "norm_bound": 1e300},
         "non-finite next evaluation", 5),
        ({"start": (0, 0), "objective": "saddle", "gradient_tol": 0}, "zero gradient", 0),
        ({"a": 0.25, "c": -2, "eta": 0.1, "gradient_tol": 0}, "iteration budget", 12),
    ]
    with localcontext() as context:
        context.prec = 80
        for settings, reason, count in cases:
            result = run(**settings)
            assert result["reason"] == reason and len(result["frames"]) == count
            x, y = (D(str(value)) for value in settings.get("start", (2, 1)))
            a, c, eta, clip = (D(str(settings.get(key, default))) for key, default in
                               (("a", 1), ("c", 0), ("eta", 0.25), ("clip", 0)))
            objective = settings.get("objective", "quadratic")

            def scalar_evaluate(px, py):
                if objective == "flat":
                    return px**4+py**4, (4*px**3, 4*py**3)
                if objective == "saddle":
                    return px**2-py**2, (2*px, -2*py)
                return a*px*px/2-a*c*px+a*c*c/2+2*py*py, (a*px-a*c, 4*py)

            for frame in result["frames"]:
                loss, (gx, gy) = scalar_evaluate(x, y)
                norm = (gx*gx+gy*gy).sqrt()
                scale = min(D(1), clip/norm) if clip > 0 else D(1)
                ux, uy = -eta*scale*gx, -eta*scale*gy
                nx, ny = x+ux, y+uy
                check(frame["point"], (x, y))
                check(frame["loss"], loss)
                check(frame["raw_gradient"], (gx, gy))
                check(frame["gradient_norm"], norm)
                check(frame["scale"], scale)
                check(frame["applied_gradient"], (scale*gx, scale*gy))
                check(frame["raw_update"], (-eta*gx, -eta*gy))
                check(frame["raw_proposal"], (x-eta*gx, y-eta*gy))
                check(frame["applied_update"], (ux, uy))
                check(frame["next"], (nx, ny))
                next_loss, next_gradient = scalar_evaluate(nx, ny)
                if frame["next_loss"] is not None:
                    check(frame["next_loss"], next_loss)
                    check(frame["next_gradient"], next_gradient)
                else:
                    assert not np.isfinite(float(next_loss))
                x, y = nx, ny

    canonical = run()
    np.testing.assert_array_equal(canonical["point"], [float(F(177147, 2097152)), 0])
    for k, frame in enumerate(canonical["frames"], 1):
        np.testing.assert_array_equal(frame["next"], [float(2*F(3, 4)**k), 0])
    loss_stop = run(gradient_tol=0, loss_tol=0.5)
    assert loss_stop["frames"][1]["next_loss"] == float(F(81, 128))
    assert abs(loss_stop["frames"][1]["loss"]-loss_stop["frames"][1]["next_loss"]) == float(F(63, 128))
    assert run(gradient_tol=0, loss_tol=0.5, iterations=2)["reason"] == "loss change tolerance"
    assert run(gradient_tol=0, eta=2, loss_tol=1e100, norm_bound=1)["reason"] == "divergence bound"
    first = run(gradient_tol=0, clip=1)["frames"][0]
    np.testing.assert_array_equal(first["raw_gradient"], [2, 4])
    np.testing.assert_array_equal(first["raw_update"], [-0.5, -1])
    np.testing.assert_array_equal(first["raw_proposal"], [1.5, 0])
    check(first["scale"], 1/np.sqrt(20.0))
    check(np.hypot(*first["applied_update"]), 0.25)
    assert run(gradient_tol=1.1, clip=1)["frames"]
    hand = run(start=(3, 1), clip=2, gradient_tol=0, iterations=1)["frames"][0]
    assert hand["gradient_norm"] == 5 and hand["scale"] == 0.4
    check(hand["applied_gradient"], [1.2, 1.6])
    check(hand["next"], [2.7, 0.6])
    assert not run(start=(0.05, 0))["frames"]
    assert not run(start=(1.5, 0), gradient_tol=1.5)["frames"]
    assert run(start=(0, 0))["reason"] == "gradient tolerance"
    assert run(eta=1e308, gradient_tol=0)["reason"] == "non-finite transition"
    assert run(start=(1e100, 0), objective="flat", gradient_tol=1e308)["reason"] == "non-finite current evaluation"
    failure = run(start=(3, 1), objective="flat", eta=2, gradient_tol=0, norm_bound=1e300)
    np.testing.assert_array_equal(failure["frames"][0]["next"], [-213, -7])
    np.testing.assert_array_equal(failure["frames"][1]["next"], [77308563, 2737])
    last = failure["frames"][-1]
    assert np.all(np.isfinite(last["next"])) and last["next_loss"] is None
    assert np.hypot(*last["next"]) < 1e300
    np.testing.assert_array_equal(failure["point"], last["point"])
    for settings in ({"clip": -1}, {"gradient_tol": -1}, {"loss_tol": -1}, {"norm_bound": 0},
                     {"iterations": 41}, {"eta": 0}, {"start": (np.nan, 0)}, {"objective": "unknown"}):
        try:
            run(**settings)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid input accepted")
    print("NumPy stop counts: gradient 11, loss 2, budget 2, clipped budget 12, divergence 8, non-finite 5, saddle 0.")
    print("All valid recorded vectors, objective values, norms, and clipping scales match independent 80-digit scalars.")
    print("Relative/absolute tolerance 1e-12; ordered reasons and counts checked exactly.")


if __name__ == "__main__":
    main()

Explore: change a rule and explain the resulting stop

Compare reset, small-loss, and budget presets. Predict the stopping state and reason before reading the final row. The last two share a point, yet only one tested a small loss difference. Disable both thresholds and observe that finite checks, divergence, and the explicit zero-gradient reason remain active.

Choose clipping and compare both arrows at the same raw gradient. Lower C, keeping the start and rate fixed, and predict the new displacement norm. Ask whether the raw norm changed; clipping occurs after the raw evaluation and must not rewrite it.

Choose divergence, then the quartic non-finite preset. Read the displayed norm bound each time: $10^6$ and $10^{300}$ serve different diagnostic demonstrations. Leaving the plotting window is not itself a stopping policy. The table retains large finite states and the failed attempt’s reason.

For a local stationary counterexample, choose the saddle $L=x^2-y^2$ at $(0,0)$ with gradient tolerance disabled. The explicit zero-gradient stop is correct. Along $(0,t)$ the loss is $-t^2<0$ , so declaring a minimum would be incorrect.

Common mistakes and counterexamples

Do not confuse the current gradient norm with the norm of an applied clipped gradient. If a raw norm is 100 and clipping sets it to 1 for the update, a raw-gradient tolerance of 2 has not been met.

A tiny loss change can reflect an excessively small step or a rounded-away update, rather than a small gradient. A small gradient can occur at a saddle or maximum. In this chapter’s saddle, the Hessian has one positive and one negative direction, so local stationarity supplies no minimum certificate.

Domain validity comes first. For $-\log x+y^2$ , a point with $x\leq0$ is outside the real objective’s domain. For $\abs x+y^2$ at $x=0$ , the classical gradient is undefined. A tolerance cannot make either required evaluation valid, and silently substituting a subgradient would introduce a different algorithm.

Numerical considerations

Finite coordinates do not guarantee a finite loss or gradient. Quartic powers can overflow while the proposed point is still representable. Check the next evaluation before subtracting losses; subtracting infinities can produce a NaN that makes a threshold comparison silently false.

Use a scaled norm calculation such as hypot rather than first squaring very large components. Retain a finite-value guard even when clipping is enabled. Clipping is defined from a valid raw gradient; it cannot recover a direction already lost to overflow or an undefined derivative.

Near a tolerance boundary, binary64 rounding can decide which side of the comparison a value occupies. The policy uses the evaluated unrounded values and inclusive thresholds. The independent reference checks a bounded collection of examples; its numerical tolerance is a verification allowance, not an extra stopping threshold.

Engineering and ML connections

A useful training record includes the objective definition, parameter scale, rate, raw and applied gradients, clipping settings, tolerances, budget, and final reason. Without those, identical-looking endpoints can conceal different stopping events or interventions.

Learning rates affect both progress and measured loss change. Constraints and penalties change feasibility or the objective; clipping instead changes an update. The explicit zero-gradient rule here belongs to plain descent: momentum or Adam can retain nonzero optimizer history even when the current raw gradient vanishes.

Exercises

State whether a test runs before or after an update. Preserve raw values when clipping is enabled, and separate a computational reason from a mathematical optimum claim.

Summary and checkpoints

Ordered diagnostics explain why plain descent stopped and what evidence remains valid. Check that you can distinguish a pre-update raw-gradient stop from a post-update loss-change stop, reproduce the clipped displacement without erasing the raw gradient, and explain why the two-step budget endpoint proves no optimum. Finite guards, divergence bounds, tolerances, and budgets each answer a specific numerical question.

What does an optimizer's stopping reason actually establish?

Inspect why this numerical run stopped

Parameter space: θ₁ and θ₂

Loss slice: θ₂ = 0

Evaluated iteration 0

Intuition: a diagnostic is evidence about a computation

Definition: two optional thresholds and one update intervention

Mechanics: the policy order is part of the meaning

Derivation: interpret each measured quantity separately

Worked examples: one path, different stopping reasons

Worked example: Inspect why this numerical run stopped

Parameter space: θ₁ and θ₂

Loss slice: θ₂ = 0

Evaluated iteration 0

Executable reference code

Explore: change a rule and explain the resulting stop

Common mistakes and counterexamples

Numerical considerations

Engineering and ML connections

Exercises

Apply the raw-gradient test before moving

Measure loss change after a finite update

Clip the update without replacing the raw gradient

Treat the budget as a limit on work

Interpret an explicit zero-gradient stop

Summary and checkpoints

The map of the book

Mathematical Language

Linear Algebra

Calculus

Probability

Statistics

Optimization

Information Theory

Numerical Computing