Mathematical Foundations

III.1 · Calculus

What value does a function approach, even where it has none?

Slide a sample point toward a, watch the values on both sides, and separate three different questions — the value at a, the left limit and the right limit.

Approach a point from both sides

Approach a = 1 on f(x) = (x² − 1)/(x − 1)

Graph of f(x) = (x² − 1)/(x − 1) near a = 1. Left samples (blue) and right samples (green) at a ∓ 1, ½, ¼, ⅛, 1/16. Left limit 2, right limit 2, f(a) is undefined. The sample point x = 1.5 has f(x) = 2.5.

Approach point
1
Sample point
1.5
Value at the sample
2.5
Actual value at a
undefinedThe denominator is exactly zero here; the expression has no value (even if it has a limit).
Left limit
2
Right limit
2
Two-sided limit
2
Distance from the limit
0.5

Exact values use =; binary64 approximations use ≈. Undefined values state their reason.

Exceptional case: The limit exists (2) but f(1) is undefined. A limit describes nearby values, not the value at the point.

lim⁡x→1−f(x)=2\lim_{x\to 1^-}f(x)=2
lim⁡x→1+f(x)=2\lim_{x\to 1^+}f(x)=2
lim⁡x→1f(x)=2\lim_{x\to 1}f(x)=2
f(1) is undefinedf(1)\ \text{is undefined}
x2−1x−1=x+1(x≠1)\frac{x^2-1}{x-1}=x+1\quad(x\ne1)
Genuine evaluations at x = a ∓ δ (never at x = a itself), a = 1
kδf(a − δ)f(a + δ)
0113
10.51.52.5
20.251.752.25
30.1251.8752.125
40.06251.93752.0625
50.031251.968752.03125
60.0156251.9843752.015625
70.00781251.99218752.0078125

Approach step 1: δ = 1

Evaluate at a − δ and a + δ, both different from a: f(a − δ) = 1, f(a + δ) = 3.

f(1−1)=1f(1+1)=3\begin{aligned}f(1-1)&=1\\ f(1+1)&=3\end{aligned}

from −2.5 to 2.5, up to 2 decimal places

from −3 to 3, up to 2 decimal places

Drag the orange sample point toward the dashed line x = a, or type x. Blue dots approach from the left, green dots from the right; the table evaluates only at x ≠ a.

Start with the hole. The values of (x2−1)/(x−1)(x^2-1)/(x-1) crowd toward 22 from both sides, yet the inspector reports f(1)f(1) as undefined: the formula divides 00 by 00 there. Then choose Value ≠ limit: now f(1)=3f(1)=3 exists, but the nearby values still approach 22. A limit is a statement about nearby values, never about the value at the point.

Intuition: watching from both sides

To ask for lim⁡x→af(x)\lim_{x\to a}f(x) is to ask: as xx gets close to aa — but stays different from it — do the values f(x)f(x) settle near one number? Approaching from the left (x<ax<a) and from the right (x>ax>a) are two separate experiments. When both settle near the same number LL, the limit is LL. When they settle on different numbers, grow without bound, or never settle, the limit does not exist, and the way it fails is itself informative.

Definition

We write lim⁡x→a−f(x)=L\lim_{x\to a^-}f(x)=L when f(x)f(x) can be made as close to LL as we like by taking x<ax<a close enough to aa; similarly for x→a+x\to a^+. The two-sided limit exists exactly when both one-sided limits exist, are finite and are equal:

lim⁡x→af(x)=L  ⟺  lim⁡x→a−f(x)=Land lim⁡x→a+f(x)=L.\begin{aligned}\lim_{x\to a}f(x)=L\iff{}&\lim_{x\to a^-}f(x)=L\\ \text{and }&\lim_{x\to a^+}f(x)=L.\end{aligned}

Precisely: for every tolerance ε>0\varepsilon>0 there is a distance δ>0\delta>0 such that 0<∣x−a∣<δ0<|x-a|<\delta implies ∣f(x)−L∣<ε|f(x)-L|<\varepsilon. The condition 0<∣x−a∣0<|x-a| excludes x=ax=a: the value f(a)f(a) — if it exists — plays no part.

Mechanics: what the explorer evaluates

The table lists f(a−δ)f(a-\delta) and f(a+δ)f(a+\delta) for δ=1,12,14,…\delta=1,\tfrac12,\tfrac14,\dots These are genuine evaluations at points different from aa; for the rational examples they are exact. The inspector’s limit rows are not read from the table. They come from the function’s exact description: a common factor (x−a)(x-a) in a rational function is divided out, the sign of an unbounded side comes from the multiplicity of the root in the denominator, and the oscillation of sin⁡(1/x)\sin(1/x) is stated analytically. The value f(a)f(a) is evaluated separately and is never replaced by the limit.

Derivation: why the hole has limit 2

For x≠1x\ne1,

x2−1x−1=(x−1)(x+1)x−1=x+1.\begin{aligned}\frac{x^2-1}{x-1}&=\frac{(x-1)(x+1)}{x-1}\\ &=x+1 .\end{aligned}

Cancelling x−1x-1 is legitimate only because x≠1x\ne1, which is exactly the situation a limit considers. The simplified expression x+1x+1 is continuous, so it approaches 1+1=21+1=2. At x=1x=1 the original expression is 0/00/0, which has no value: the function has a removable hole.

For xsin⁡(1/x)x\sin(1/x) there is no cancellation, but a squeeze works: ∣xsin⁡(1/x)∣≤∣x∣|x\sin(1/x)|\le|x| for x≠0x\ne0, and ∣x∣→0|x|\to0. So the limit is 00 even though sin⁡(1/x)\sin(1/x) alone has no limit.

Worked example

Take f(x)=(x2−1)/(x−1)f(x)=(x^2-1)/(x-1) and a=1a=1. With δ=1\delta=1: f(0)=1f(0)=1 and f(2)=3f(2)=3. With δ=12\delta=\tfrac12: f(0.5)=1.5f(0.5)=1.5 and f(1.5)=2.5f(1.5)=2.5. With δ=14\delta=\tfrac14: 1.751.75 and 2.252.25. Each pair sits exactly δ\delta away from 22, because f(x)−2=x−1f(x)-2=x-1. The left and right limits are both 22, so lim⁡x→1f(x)=2\lim_{x\to1}f(x)=2, while f(1)f(1) is undefined.

Replay the worked derivation

Approach step 1: δ = 1

Evaluate at a − δ and a + δ, both different from a: f(a − δ) = 1, f(a + δ) = 3.

f(1−1)=1f(1+1)=3\begin{aligned}f(1-1)&=1\\ f(1+1)&=3\end{aligned}
Replay the approach: each step evaluates both sides at distance δ, then the limit is decided analytically.

Failure cases

  • Jump. For the step function, the left limit at 00 is −1-1 and the right limit is 11. They differ, so no two-sided limit exists.
  • Unbounded. 1/x21/x^2 grows without bound from both sides of 00; 1/x1/x goes to −∞-\infty from the left and +∞+\infty from the right. Writing lim⁡=+∞\lim=+\infty records how the limit fails; ∞\infty is not a real number.
  • Oscillation. sin⁡(1/x)\sin(1/x) reaches both −1-1 and 11 on every interval around 00. Along x=1/(kπ)x=1/(k\pi) the values are 00; along x=2/((4k+1)π)x=2/((4k+1)\pi) they are 11.

Executable reference code

The pedagogical file evaluates exact one-sided tables with Fraction, cancels the common factor and shows the jump. The NumPy file samples two sequences of sin⁡(1/x)\sin(1/x) that approach 00 and disagree, and checks the squeeze bound for xsin⁡(1/x)x\sin(1/x).

code/python/calculus_limits.py
"""One-sided approach tables with exact Fractions (III.1). We evaluate only at x != a; f(a) is a separate question."""
from fractions import Fraction as F


def hole(x):
    if x == 1:
        raise ZeroDivisionError("(x**2 - 1)/(x - 1) is 0/0 at x = 1: undefined")
    return (x * x - 1) / (x - 1)


def approach(f, a, steps=8):
    rows = []
    for k in range(steps):
        d = F(1, 2**k)
        rows.append((d, f(a - d), f(a + d)))
    return rows


rows = approach(hole, F(1))
# f(x) - 2 = x - 1 exactly, so each sample sits exactly delta away from 2.
assert all(left == 2 - d and right == 2 + d for d, left, right in rows)

# The value at the point is not the limit: it does not even exist.
try:
    hole(F(1))
except ZeroDivisionError:
    pass
else:
    raise AssertionError("f(1) must be undefined")

# A jump: the two one-sided tables settle on different numbers.
step = lambda x: F(-1) if x < 0 else F(1)
jump = approach(step, F(0))
assert {left for _, left, _ in jump} == {-1} and {right for _, _, right in jump} == {1}
code/python/calculus_limits_numpy.py
"""Two approaches to 0 that disagree for sin(1/x), and a squeeze that works for x*sin(1/x) (III.1)."""
import numpy as np

k = np.arange(1, 200)
zeros = 1 / (k * np.pi)                  # sin(1/x) = sin(k*pi) = 0
ones = 2 / ((4 * k + 1) * np.pi)         # sin(1/x) = sin(pi/2 + 2k*pi) = 1
assert np.allclose(np.sin(1 / zeros), 0, atol=1e-12)
assert np.allclose(np.sin(1 / ones), 1, atol=1e-12)
# Both sequences approach 0 from the right, yet the values approach 0 and 1: no limit exists.
assert zeros[-1] < 1e-2 and ones[-1] < 1e-2

# Squeeze: |x sin(1/x)| <= |x| -> 0.
x = np.concatenate([-np.logspace(-1, -8, 400), np.logspace(-8, -1, 400)])
assert np.all(np.abs(x * np.sin(1 / x)) <= np.abs(x))

Explore

Move aa away from the special point on each function: away from its trouble spot every example is continuous and its limit equals its value. On 1/x, compare the blue and green columns. On x·sin(1/x), watch the distance row shrink as the sample approaches 00.

Common mistakes

Substituting x=ax=a answers a different question — the value, not the limit — and can produce 0/00/0. Agreement of a few table entries is not a proof. A limit that is “infinite” does not exist as a real number. A two-sided limit needs both sides; one convincing side is not enough.

Numerical considerations

Floating-point evaluation of (x2−1)/(x−1)(x^2-1)/(x-1) very close to 11 suffers cancellation: both numerator and denominator lose relative accuracy. The exact rational table avoids this; a binary64 table would eventually show noise. Choosing a sequence of points also chooses what you can see: aliasing and cancellation can make a numerical table misleading in either direction.

Engineering and ML connections

Limits underlie every derivative (III.3) and integral (III.12). In computation they appear as questions of stability: does a quantity settle as a step size shrinks, a learning rate decreases or a sample size grows? Expressions such as 0/00/0 appear in normalisations (dividing by a vanishing norm), and robust code handles them explicitly rather than hoping for a limiting value.

Exercises

calculation

Cancel, then approach

Find the limit of (x² − 4)/(x − 2) as x → 2.

calculation

Read both sides

f(x) = −1 for x < 0 and f(x) = 2x + 1 for x ≥ 0. Give the left and right limits at 0.

prediction

Value versus limit

g(x) = x + 1 for x ≠ 1 and g(1) = 3. What is the limit as x → 1?

Choose one answer

prediction

An unbounded approach

What is the correct statement about 1/x² as x → 0?

Choose one answer

prediction

Trust a table?

A table of sin(1/x) at x = 1/(kπ) shows only zeros. Does sin(1/x) → 0 as x → 0?

Choose one answer

Summary and checkpoints

A limit describes the values near aa, from each side, not the value at aa. It exists when both one-sided limits exist, are finite and agree. Removable holes, jumps, unbounded and oscillating behaviour are distinct ways to fail. Next, III.2 asks when the value at a point does equal its limit: continuity.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available