Mathematical Foundations

II.7 · Linear Algebra

When do two directions give unique coordinates?

Hold a target fixed and move the two basis vectors. Its coordinates change, its position does not, and singular directions reveal the difference between reachability and uniqueness.

Move the basis; keep the target fixed

These two directions form a basis. Each target has exactly one coordinate pair (a, b); the target stays fixed when you move only the basis.

first direction
(2, 1)
second vector
(1, 2)
target in standard coordinates
(3, 0)
two-direction determinant
3
first basis coordinate
2
second basis coordinate
−1
reconstructed target
(3, 0)
relative separation of directions
0.600000

Exact expressions use =. Rounded computations use ≈. All arrows use these same quantities.

These two directions form a basis. Each target has exactly one coordinate pair (a, b); the target stays fixed when you move only the basis.

Δ=uxvy−uyvx=3≠0\Delta=u_xv_y-u_yv_x=3\ne0
a=pxvy−pyvxΔ=2,b=uxpy−uypxΔ=−1a=\frac{p_xv_y-p_yv_x}{\Delta}=2,\quad b=\frac{u_xp_y-u_yp_x}{\Delta}=-1
p=2[21]+(−1)[12]=[30]\mathbf p=2\begin{bmatrix}2\\1\end{bmatrix}+(-1)\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}3\\0\end{bmatrix}

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

Move u or v while keeping target p fixed. The readout solves p = au + bv exactly. Move p to test a new target. Collinear directions remove unique coordinates, and almost-collinear directions reveal sensitivity.

Start with target (3,0)(3,0). Predict whether its two weights change when you move only u\mathbf u. Then choose Collinear directions: the target remains where it was, but the unique coordinate rows become undefined. Move the target onto that line to see why being reachable still does not produce a unique pair.

Intuition: coordinates depend on a measuring frame

The standard components (3,0)(3,0) describe a point relative to one horizontal and one vertical unit direction. A different pair of directions can describe the same target with different weights. Coordinates are instructions for reconstructing a vector from a chosen basis; they are not an intrinsic pair attached to the target independently of the frame.

A useful basis supplies enough directions to reach every target and avoids redundant directions that would give multiple descriptions. The stage holds p\mathbf p fixed when a basis tip moves, so the changing coordinates reveal a change of description rather than a motion of the target.

Definition

Two vectors are linearly independent when

au+bv=0⟹a=b=0.a\mathbf u+b\mathbf v=\mathbf0 \quad\Longrightarrow\quad a=b=0.

A basis of R2\mathbb R^2 is an ordered pair of independent vectors that spans the plane. In two dimensions, two independent vectors already span the plane. For a chosen ordered basis (u,v)(\mathbf u,\mathbf v), the coordinates of p\mathbf p are the unique pair (a,b)(a,b) satisfying

p=au+bv.\mathbf p=a\mathbf u+b\mathbf v.

Order matters: swapping the basis vectors swaps their coordinate roles. Orthogonality and unit lengths are convenient but are not required. A slanted, unequal-length independent pair is still a basis.

Mechanics: reconstruction and singularity

The blue and green basis arrows show the two directions. The head-to-tail weighted arrows reconstruct the target. Grid lines correspond to fixed basis coefficients, so moving the basis changes the coordinate grid. The inspector keeps the standard target components separate from the solved basis coordinates.

The stage’s independence test is the exact two-dimensional quantity

Δ=uxvy−uyvx.\Delta=u_xv_y-u_yv_x.

This is the determinant of the two basis columns. Here it is a practical independence test; its full area, orientation, and higher-dimensional interpretation belongs to II.12. If Δ=0\Delta=0, the pair is singular and cannot be a basis of the plane. The stage then tests whether the target lies in the surviving span, and distinguishes no representation from infinitely many representations. It never supplies a fabricated unique coordinate pair.

Derivation: an independent pair has unique weights

The component equations are

uxa+vxb=px,uya+vyb=py.u_xa+v_xb=p_x,\qquad u_ya+v_yb=p_y.

Multiply the first by vyv_y, the second by vxv_x, and subtract. The bb terms cancel, leaving Δa=pxvy−pyvx\Delta a=p_xv_y-p_yv_x. Eliminate aa similarly. For Δ≠0\Delta\ne0,

a=pxvy−pyvxΔ,b=uxpy−uypxΔ.a=\frac{p_xv_y-p_yv_x}{\Delta},\qquad b=\frac{u_xp_y-u_yp_x}{\Delta}.

These formulas give a representation for every target. To prove uniqueness, suppose another pair (a′,b′)(a',b') gives the same target. Subtract the two representations: (a−a′)u+(b−b′)v=0(a-a')\mathbf u+(b-b')\mathbf v=\mathbf0. Independence forces both differences to vanish.

If the two directions are collinear, a nonzero combination can cancel them. For instance, v=2u\mathbf v=2\mathbf u gives 2u−v=02\mathbf u-\mathbf v=\mathbf0, proving dependence. Any reachable target then has infinitely many representations because adding a multiple of this cancelling pair changes the coefficients without changing the vector. A zero basis vector immediately creates such redundancy.

Worked examples

For u=(2,1)\mathbf u=(2,1), v=(1,2)\mathbf v=(1,2), and p=(3,0)\mathbf p=(3,0), the determinant is 33. The formulas yield a=2a=2 and b=−1b=-1, checked by 2(2,1)−(1,2)=(3,0)2(2,1)-(1,2)=(3,0). The target’s standard components (3,0)(3,0) and its basis coordinates (2,−1)(2,-1) describe the same vector in different frames.

Inspect the worked calculation

For u = (2, 1), v = (1, 2), p = (3, 0), Δ = 4 − 1 = 3.

Δ=uxvy−uyvx=3≠0\Delta=u_xv_y-u_yv_x=3\ne0
first direction
(2, 1)
second vector
(1, 2)
target in standard coordinates
(3, 0)
two-direction determinant
3
first basis coordinate
2
second basis coordinate
−1
reconstructed target
(3, 0)
relative separation of directions
0.600000
Inspect the determinant, solve the two coordinates, and check reconstruction.

For u=(1,1)\mathbf u=(1,1) and v=(2,2)\mathbf v=(2,2), target (3,3)(3,3) is reachable with any a+2b=3a+2b=3, including (3,0)(3,0) and (1,1)(1,1). Target (3,0)(3,0) is unreachable because every combination has equal components. A singular pair can therefore fail through nonuniqueness or through nonexistence, depending on the target.

Executable reference code

The pedagogical code implements the elimination formulas with exact decimal fractions and checks reconstruction. The NumPy example forms an array with basis vectors as columns and solves for the weights. This small practical bridge follows the book’s column-vector convention; a general systems chapter is still ahead. A singular solve raises an error rather than returning unique coordinates.

code/python/linear_basis.py
from fractions import Fraction


def coordinates(u, v, p):
    if any(len(w) != 2 for w in (u, v, p)):
        raise ValueError('Expected two components per vector')
    u, v, p = (tuple(Fraction(str(x)) for x in w) for w in (u, v, p))
    determinant = u[0]*v[1] - u[1]*v[0]
    if determinant == 0:
        raise ValueError('Singular directions: unique coordinates do not exist')
    a = (p[0]*v[1] - p[1]*v[0]) / determinant
    b = (u[0]*p[1] - u[1]*p[0]) / determinant
    assert tuple(a*x + b*y for x, y in zip(u, v)) == p
    return a, b


if __name__ == '__main__':
    assert coordinates((2, 1), (1, 2), (3, 0)) == (2, -1)
code/python/linear_basis_numpy.py
import numpy as np

u = np.array([2.0, 1.0])
v = np.array([1.0, 2.0])
p = np.array([3.0, 0.0])
columns = np.column_stack((u, v))
# solve raises LinAlgError for a singular matrix; never invent coordinates.
coordinates = np.linalg.solve(columns, p)
reconstructed = coordinates[0] * u + coordinates[1] * v
assert np.allclose(coordinates, [2, -1])
assert np.allclose(reconstructed, p)
# For nearly dependent columns, check sensitivity as well as the residual.
condition = np.linalg.cond(columns)

Explore

Try the standard basis (1,0),(0,1)(1,0),(0,1) and confirm that the weights match the target’s components. Then double only the first basis vector: the first weight halves while the target stays fixed. Swap the two basis vectors with numeric entry and inspect the swapped coordinate roles. Make one basis vector zero, then both zero, and compare which targets remain reachable.

Choose Almost collinear. The exact determinant is 0.00010.0001, so the pair remains independent. Reconstructing a target away from the narrow direction can require very large weights of opposite signs. The plane fits those actual weighted arrows; the small target may consequently occupy very little screen space. Reset returns to a comfortably separated pair.

Common mistakes

Two nonzero vectors can still be dependent. Two perpendicular nonzero vectors are independent, but perpendicularity is sufficient rather than necessary. A reachable target does not prove the pair spans the plane, and a small reconstruction residual alone does not prove the computed coordinates are accurate. Also distinguish basis change from moving the target: changing the measuring frame changes the numbers describing the same vector.

Numerical considerations

Exact independence asks whether Δ\Delta is zero. Numerical reliability asks how sensitive the coordinates are to errors in the vectors and target. The stage reports the relative directional separation ∣sin⁡θ∣|\sin\theta|, which removes the vectors’ overall scale. Below 10−410^{-4} it warns about sensitivity; this warning never changes exact independence. The threshold is a teaching cue for this bounded scene, not a general solver’s rank decision.

Near dependence, large weighted moves nearly cancel. Rounding can then alter the final target substantially, even when each large move changes only slightly. NumPy’s condition estimate previews this issue; practical rank and solver tolerances require the scale and uncertainty of actual data. Exact arithmetic on entered decimals certifies the intended reconstruction but cannot repair inaccurate measurements. The stage rejects invalid numeric input and shows fractions exactly, while rounded decimal companions use ≈\approx.

Engineering and ML connections

A basis is a coordinate system for signals, motions, or features. Redundant directions can create multiple parameter settings with the same output; changing a basis changes the weights without necessarily changing the represented object. The column convention also prepares a transformation view of matrices: specify where the standard basis directions go, and every other vector follows by the same weighted-combination rule.

Exercises

calculation

A new coordinate pair

u = (2, 1), v = (1, 2). Find (a, b) for p = (3, 0).

calculation

Test two directions

u = (1, 2), v = (2, 4). Give Δ = uxvy − uyvx.

construction

Construct from coordinates

u = (2, 1), v = (1, 2). Find p from basis coordinates (−1, 2).

prediction

A singular configuration

u = (1, 1), v = (2, 2), p = (3, 3). How many coefficient pairs represent p?

Choose one answer

explanation

Almost collinear

Δ is nonzero but the directions almost agree. What is true?

Choose one answer

Summary and checkpoints

Independent directions supply unique weights; a basis makes those weights coordinates. Check that you can derive the two-coordinate formulas, distinguish unreachable from nonunique singular targets, and explain why exact independence can coexist with numerical sensitivity. You are ready for II.8, where a matrix will describe a transformation by its effect on basis directions.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available