What does one number tell us about a whole parameter choice?

On the default quadratic, compare $(2,0)$ with $(0,1)$ before reading the values. Both have loss $2$. Do they also have the same sensitivity to a small parameter change? Move to $(2,1)$ and recover loss $4$ from its two separate contributions.

Intuition: a score attached to every candidate

An engineer choosing two controller gains needs a way to compare settings. A loss might combine tracking error with the energy used by the controller. Each parameter pair is one candidate; the objective assigns it one scalar score. Smaller scores are preferred when we minimize.

The drawn landscape places parameter coordinates on the horizontal plane and objective values at their heights. A contour joins points with equal score. It is not an optimizer path: points on the same contour can have different gradients, and an algorithm can cross many contours in one update. The gradient describes local sensitivity; curvature describes how that sensitivity changes.

Definition: objective, domain, and optimizer

An objective function is a map $L:D\to\R$ from a specified parameter domain $D$. Here $\theta=(x,y)$ abbreviates a two-component column vector. Minimization asks for the smallest attainable value, and for the parameter choices attaining it:

This notation assumes a minimum exists. The infimum can describe a lower bound without attainment, and an objective can instead be unbounded below. The scalar $L^*$ and the set of minimizing parameters are different objects. An optimizer is an algorithm searching for parameters; it is not the objective itself.

Our recurring family is

For $a>0$ its unique minimizer is $(c,0)$. The explorer keeps $a>0$. In the analytic extensions used by the code and exercises, $a=0$ makes every $(x,0)$ a minimizer, while $a<0$ makes the loss unbounded below along the x direction. The parameters $a,c$ describe the chosen landscape, while $x,y$ are the parameters being optimized within that landscape.

Mechanics: one point, several derived quantities

Read the current point $\theta$, its loss $L(\theta)$, and its gradient together. Differentiate the gradient to obtain the Hessian used in the analysis below. Changing $x$ or $y$ evaluates a different candidate. Changing $c$ moves the quadratic’s horizontal center. Changing $a$ changes horizontal curvature; it does not change the vertical curvature $4$.

The default equations are

A negative gradient component means increasing that coordinate lowers the loss to first order. A positive component means decreasing it lowers the loss to first order. Neither statement yet chooses a finite distance to travel. Keep the plotted point, the inspector’s coordinates, and the formula substitution in agreement.

Derivation: contours and the local model

Differentiate each term of the family independently:

Expanding the squares gives an exact identity for a displacement $\delta=(u,v)$:

The gradient term is the linear prediction of the loss change; the Hessian term accounts for the bending of this quadratic. For a general smooth function this expression is a local second-order approximation, with a remainder. For this family it is exact at every displacement.

For $a>0$ and a positive contour level $\ell$, rearrange $L=\ell$:

These ellipses have horizontal radius $\sqrt{2\ell/a}$ and vertical radius $\sqrt{\ell/2}$. At the default $a=1$, equal horizontal and vertical parameter displacements have different costs. The fourfold vertical curvature produces the narrower vertical radius.

Worked examples: equal height does not mean equal slope

At $(2,1)$ with $a=1,c=0$,

The point $(2,0)$ has loss $2$ and gradient $(2,0)$. The point $(0,1)$ also has loss $2$, but gradient $(0,4)$. Their gradient norms are $2$ and $4$: equal heights on a contour do not give equal local steepness. A small displacement in the negative y direction from the second point changes loss twice as fast per unit distance as steepest descent from the first.

Now choose $a=2,c=1$ and $\theta=(2,-1/2)$. The displacement from the center is $x-c=1$, so

Executable reference code

The standard-library reference uses Fraction for the polynomial objectives, gradients, and the quadratic Hessian. It independently checks literal points on equal contours and the contrasting landscapes. The logarithmic example is separate: real logarithms are evaluated as binary64 approximations, with an explicit $x>0$ domain check.

The NumPy program evaluates a batch of points through a quadratic form and matrix multiplication, cross-checks the Hessian spectrum with eigvalsh, and samples a double-well grid with array operations. Finding the smallest sampled grid value is evidence about those samples; it is not a proof of a whole-domain minimum.

"""Exact polynomial landscapes; logarithmic values use binary64 separately."""
from fractions import Fraction as F
from math import isfinite, log


def quadratic(x, y, a=1, c=0):
    x, y, a, c = map(F, (x, y, a, c))
    loss = a * (x - c)**2 / 2 + 2 * y**2
    gradient = (a * (x - c), 4 * y)
    hessian = ((a, F(0)), (F(0), F(4)))
    return loss, gradient, hessian


def polynomial_losses(x, y):
    x, y = F(x), F(y)
    return {
        "double well": (x*x - 1)**2 + y*y,
        "cubic": x**3 - 3*x + y*y,
        "ridge": -x*x - y*y,
        "saddle": x*x - y*y,
        "flat": x**4 + y**4,
        "degenerate": x**3 + y*y,
        "nonsmooth": abs(x) + y*y,
    }


def barrier(x, y):
    x, y = float(x), float(y)
    if not isfinite(x) or not isfinite(y) or x <= 0:
        raise ValueError("The log objective requires finite x > 0 and finite y")
    loss, gradient = -log(x) + y*y, (-1/x, 2*y)
    if not all(isfinite(v) for v in (loss, *gradient)):
        raise OverflowError("The binary64 result is non-finite")
    return loss, gradient


def main():
    loss, gradient, hessian = quadratic(2, 1)
    assert loss == 4 and gradient == (2, 4)
    assert hessian == ((1, 0), (0, 4))
    assert quadratic(2, 1, a=2, c=1)[:2] == (3, (2, 4))
    assert quadratic(0, 1)[0] == 2 and quadratic(2, 0)[0] == 2
    assert polynomial_losses(-1, 0)["double well"] == 0
    assert polynomial_losses(1, 0)["double well"] == 0
    assert polynomial_losses(0, 0)["double well"] == 1
    assert polynomial_losses(1, 0)["cubic"] == -2
    assert polynomial_losses(-3, 0)["cubic"] == -18
    assert polynomial_losses(1, 0)["ridge"] == -1
    assert polynomial_losses(1, 0)["saddle"] == 1
    assert polynomial_losses(0, 1)["saddle"] == -1
    assert polynomial_losses(0, 0)["flat"] == 0
    assert polynomial_losses(-1, 0)["degenerate"] == -1
    h = F(1, 100)
    right = polynomial_losses(h, 0)["nonsmooth"] / h
    left = polynomial_losses(-h, 0)["nonsmooth"] / -h
    assert (left, right) == (-1, 1)  # No classical x derivative at zero.
    assert barrier(1, 0) == (0.0, (-1.0, 0.0))
    for x in (0, -1, float("nan"), float("inf")):
        try:
            barrier(x, 0)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid logarithmic domain accepted")
    print("Exact canonical loss, gradient, Hessian:", loss, gradient, hessian)
    print("Exact alternative losses at (1,0):", polynomial_losses(1, 0))
    print("Log objective at (2,1), binary64 approximation:", barrier(2, 1))


if __name__ == "__main__":
    main()

"""Vectorized binary64 landscape samples; a finite grid is not an optimum proof."""
import numpy as np


def quadratic(points, a=1.0, c=0.0):
    points = np.asarray(points, dtype=np.float64)
    if points.ndim != 2 or points.shape[1] != 2 or not np.all(np.isfinite(points)):
        raise ValueError("Require finite points of shape (n,2)")
    if not np.isfinite(a) or not np.isfinite(c):
        raise ValueError("Require finite a and c")
    shifted = points - np.array([c, 0.0])
    hessian = np.diag([a, 4.0])
    loss = np.einsum("ni,ij,nj->n", shifted, hessian, shifted) / 2
    gradient = shifted @ hessian
    if not np.all(np.isfinite(loss)) or not np.all(np.isfinite(gradient)):
        raise OverflowError("The binary64 result is non-finite")
    return loss, gradient, hessian


def barrier(points):
    points = np.asarray(points, dtype=np.float64)
    if points.ndim != 2 or points.shape[1] != 2 or not np.all(np.isfinite(points)) or np.any(points[:, 0] <= 0):
        raise ValueError("Require finite (n,2) points with x > 0")
    x, y = points.T
    loss = -np.log(x) + y*y
    gradient = np.column_stack((-1/x, 2*y))
    if not np.all(np.isfinite(loss)) or not np.all(np.isfinite(gradient)):
        raise OverflowError("The binary64 result is non-finite")
    return loss, gradient


def main():
    points = np.array([[2, 1], [2, 0], [0, 1], [0, 0]], dtype=np.float64)
    loss, gradient, hessian = quadratic(points)
    np.testing.assert_array_equal(loss, [4, 2, 2, 0])
    np.testing.assert_array_equal(gradient, [[2, 4], [2, 0], [0, 4], [0, 0]])
    np.testing.assert_array_equal(np.linalg.eigvalsh(hessian), [1, 4])
    np.testing.assert_array_equal(quadratic([[2, 1]], a=2, c=1)[0], [3])
    coordinates = np.linspace(-2, 2, 17)
    x, y = np.meshgrid(coordinates, coordinates, indexing="xy")
    wells = (x*x - 1)**2 + y*y
    assert wells.min() == 0 and np.count_nonzero(wells == 0) == 2
    np.testing.assert_array_equal(barrier([[1, 0]])[1], [[-1, 0]])
    for invalid in ([[0, 0]], [[-1, 0]], [[np.nan, 0]]):
        try:
            barrier(invalid)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid logarithmic domain accepted")
    print("Quadratic sample losses ≈", loss, "gradients ≈", gradient)
    print("Binary64 grid minimum ≈", wells.min(), "(finite samples only)")


if __name__ == "__main__":
    main()

Explore: compare different mechanisms

Use the alternative landscapes as questions about the objective, before choosing an algorithm:

Predict the change along one coordinate, then inspect another. On the quadratic, increase $a$ while keeping $c$ fixed: which contours become narrower? Move $c$ without changing $a$: which derivatives change at a fixed $(x,y)$, and which second derivatives remain fixed?

Common mistakes and counterexamples

A loss is not automatically a probability, a nonnegative quantity, or a distance. The cubic and logarithmic objectives legitimately have negative values. A zero objective value alone says nothing about optimality: $-\log x+y^2$ is zero at $(1,0)$ and lower at larger x with $y=0$.

Changing the plotted vertical scale can disguise an actual change to the objective. If $\widetilde L=3L+7$, all rankings and minimizers stay the same, but $\nabla\widetilde L=3\nabla L$. An algorithm using the same learning rate now takes a triple displacement. The constant $7$ changes displayed heights but contributes no derivative.

The displayed rectangle is also not the mathematical domain. A low point on the edge of a finite picture may simply be the lowest visible sample of an unbounded objective. Inspect the formula and its domain before declaring a winner.

Numerical considerations

Exact rational substitutions in the polynomial examples justify equality signs. Logarithms, sampled contour geometry, and numerical trajectories use binary64 approximations; displayed rounding must not become an exact mathematical claim. Rounding two losses to the same visible number does not establish that the underlying values are equal.

Large parameter magnitudes can overflow squared or higher-order terms even when inputs are finite. The references check the logarithmic domain and reject non-finite floating results where they expose floating inputs. A gap in a plot may represent an invalid domain point rather than a very high loss. Near $x=0$ the logarithmic gradient $-1/x$ also becomes large, so domain safety matters for updates as well as for display.

Engineering and ML connections

A training objective often aggregates data-fitting errors and parameter penalties. Choosing a sum instead of a mean multiplies the data term by the number of examples. With no other terms that preserves minimizers but changes gradient scale; when a separate penalty is retained, it also changes their relative weight. State the entire objective before interpreting an optimizer trace.

Least squares gives concrete quadratic fitting objectives. The gradient chapter explains directional sensitivity, and Hessian and curvature explains local shape. These become the ingredients of the parameter updates in gradient descent. A better search algorithm cannot repair an objective that rewards the wrong engineering behavior.

Exercises

Keep the objective value, its derivatives, and the domain distinct. Use exact fractions for the hand calculations.

Summary and checkpoints

An objective maps a valid parameter choice to a scalar score. Contours preserve score; gradients describe local changes; Hessians describe curvature. Check that you can recover loss $4$, gradient $(2,4)$, and Hessian $\operatorname{diag}(1,4)$ at $(2,1)$, distinguish changing a point from changing a landscape, and reject an out-of-domain logarithmic input. Next, compare a nearby best point with a best point over the whole domain.