Mathematical Foundations

II.6 · Linear Algebra

What can two weighted moves reach?

Change two coefficients and watch the destination move through a parallelogram. Change the directions themselves and see the reachable set collapse from plane to line to point.

Two weights explore the reachable set

Unrestricted real coefficients reach the entire plane. The shaded parallelogram is only the coefficient window −2 ≤ a,b ≤ 2; its grid comes from these actual directions.

first direction
(2, 1)
second vector
(1, 2)
first weight
1
second weight
−0.5
linear combination
(1.5, 0)
number of independent directions
2

Exact expressions use =. Rounded computations use ≈. All arrows use these same quantities.

Unrestricted real coefficients reach the entire plane. The shaded parallelogram is only the coefficient window −2 ≤ a,b ≤ 2; its grid comes from these actual directions.

w=1[21]+(−0.5)[12]=[1.50]\mathbf w=1\begin{bmatrix}2\\1\end{bmatrix}+(-0.5)\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}1.5\\0\end{bmatrix}
span⁡{u,v}=R2\operatorname{span}\{\mathbf u,\mathbf v\}=\mathbb R^2

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −2 to 2, up to 2 decimal places

from −2 to 2, up to 2 decimal places

Change a and b to move w = au + bv. Drag either direction to change the reachable geometry. The shaded parallelogram is the image of the bounded coefficient square −2 ≤ a,b ≤ 2; unrestricted coefficients define the span.

Move just aa, then just bb, and identify the two directions of travel. Choose Collinear directions: two controls still work, but their effects now lie on the same line. Choose Both directions zero: changing either weight cannot move the result. The geometry follows the actual vectors in every case.

Intuition: directions are resources

One nonzero vector supplies a line of possible moves, because its scalar multiples point along that line in either direction. Two vectors can supply two different directional resources. A weighted sum first scales each move, then adds them head-to-tail. Independent directions let the two weights navigate the plane.

Two arrows do not necessarily supply two directions. If one is twice the other, changing its coefficient only changes how far you move along a line already available. Zero contributes no directional resource at all. This is why the number of controls is not the dimension of the reachable set.

Definition

For real coefficients a,ba,b, a linear combination is

w=au+bv.\mathbf w=a\mathbf u+b\mathbf v.

The span is the set of all such combinations with unrestricted real coefficients:

span⁡{u,v}={au+bv:a,b∈R}.\operatorname{span}\{\mathbf u,\mathbf v\} =\{a\mathbf u+b\mathbf v:a,b\in\mathbb R\}.

In the plane it can be all of R2\mathbb R^2, a line through the origin, or the singleton {0}\{\mathbf0\}. It always contains zero, obtained with a=b=0a=b=0. It is also closed under addition and scalar multiplication. These properties make it a vector subspace; a translated line that misses the origin is not a span of ordinary vectors.

Mechanics: a coefficient square becomes a parallelogram

For fixed directions, varying aa while holding bb fixed traces a line parallel to u\mathbf u. Varying bb traces a line parallel to v\mathbf v. The stage’s coefficient grid uses several fixed values of each weight. Every line endpoint is computed as an actual combination of the current vectors.

The corners of the bounded region are −2u−2v-2\mathbf u-2\mathbf v, 2u−2v2\mathbf u-2\mathbf v, 2u+2v2\mathbf u+2\mathbf v, and −2u+2v-2\mathbf u+2\mathbf v. These are the images of the four corners of the control square. When the directions become collinear, the corners and grid collapse onto a segment. When both become zero, they collapse to the origin. Nothing remains shaded as a plane when the mathematics has only one or zero directions.

Derivation: plane, line, or point

Suppose v=cu\mathbf v=c\mathbf u with u≠0\mathbf u\ne0. Then

au+bv=(a+bc)u.a\mathbf u+b\mathbf v=(a+bc)\mathbf u.

Every result lies on u\mathbf u‘s line, and every point on that line is reachable by choosing aa appropriately. If both vectors are zero, every combination is zero. If one is zero and the other is nonzero, the nonzero vector still spans a line.

When neither direction is a multiple of the other, every target in the plane can be written as a combination. To see the mechanism, begin by moving along u\mathbf u. The second direction supplies a nonzero sideways component relative to that line; choose bb to match the target’s sideways displacement, then choose aa to correct the remaining parallel displacement. The next chapter turns this argument into exact coordinate formulas and proves uniqueness.

Closure under addition follows by grouping weights:

(au+bv)+(cu+dv)=(a+c)u+(b+d)v.(a\mathbf u+b\mathbf v)+(c\mathbf u+d\mathbf v) =(a+c)\mathbf u+(b+d)\mathbf v.

Scaling a combination likewise scales both weights. These identities explain why linear combinations remain inside the same span.

Worked examples

With u=(2,1)\mathbf u=(2,1), v=(1,2)\mathbf v=(1,2), a=1a=1, and b=−0.5b=-0.5, the weighted moves are (2,1)(2,1) and (−0.5,−1)(-0.5,-1). Their sum is (1.5,0)(1.5,0), despite neither original vector lying on the horizontal axis.

Inspect the worked calculation

With u = (2, 1), v = (1, 2), a = 1 and b = −0.5, the two moves are (2, 1) and (−0.5, −1).

w=1[21]+(−0.5)[12]=[1.50]\mathbf w=1\begin{bmatrix}2\\1\end{bmatrix}+(-0.5)\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}1.5\\0\end{bmatrix}
first direction
(2, 1)
second vector
(1, 2)
first weight
1
second weight
−0.5
linear combination
(1.5, 0)
number of independent directions
2
Inspect the two scaled moves and their sum; the negative weight matters.

For u=(1,2)\mathbf u=(1,2) and v=(2,4)\mathbf v=(2,4), the combination reduces to (a+2b)(1,2)(a+2b)(1,2). Both (a,b)=(1,0)(a,b)=(1,0) and (−1,1)(-1,1) produce (1,2)(1,2). A target such as (1,0)(1,0) is impossible because every reachable point satisfies y=2xy=2x. This is a failed construction for a mathematical reason, independent of the control range.

Executable reference code

The pedagogical code spells out the weighted component sums and identifies zero, one, or two directions. NumPy shows the same combination directly, then maps a small table of coefficient pairs into sample points. Those samples illustrate a bounded region; a finite sample table is never the entire continuous span.

code/python/linear_span.py
from fractions import Fraction


def combine(a, u, b, v):
    if len(u) != 2 or len(v) != 2:
        raise ValueError('Expected two components per vector')
    a, b = Fraction(str(a)), Fraction(str(b))
    return tuple(a*Fraction(str(x)) + b*Fraction(str(y)) for x, y in zip(u, v))


def span_dimension(u, v):
    if len(u) != 2 or len(v) != 2:
        raise ValueError('Expected two components per vector')
    u, v = (tuple(Fraction(str(x)) for x in w) for w in (u, v))
    cross = u[0]*v[1] - u[1]*v[0]
    return 2 if cross else 1 if any(u) or any(v) else 0


if __name__ == '__main__':
    assert combine(1, (2, 1), '-0.5', (1, 2)) == (Fraction('1.5'), 0)
    assert span_dimension((1, 1), (2, 2)) == 1
    assert span_dimension((0, 0), (0, 0)) == 0
code/python/linear_span_numpy.py
import numpy as np

u = np.array([2.0, 1.0])
v = np.array([1.0, 2.0])
a, b = 1.0, -0.5
w = a * u + b * v
assert np.array_equal(w, [1.5, 0])
# Sample the bounded coefficient square, not the entire mathematical span.
coefficients = np.array([[a, b] for a in [-2, 0, 2] for b in [-2, 0, 2]])
columns = np.column_stack((u, v))
reachable_samples = coefficients @ columns.T
assert reachable_samples.shape == (9, 2)

Explore

Set a=0a=0 and vary bb: only the second direction contributes. Set b=0b=0 and vary aa. Use opposite signs to reach points beyond the wedge between the positive arrows. Then change v\mathbf v until it lies on u\mathbf u‘s line; explain why the two families of grid lines no longer cross in distinct directions. Test a zero first vector and a nonzero second vector before setting both to zero.

In the almost-collinear preset, the region is very thin but remains two-dimensional. Exact entered decimals retain two distinct directions. Targets far from that thin strip may require large unrestricted weights; being a plane-spanning pair does not guarantee that the current control range can reach a chosen target.

Common mistakes

Restricting both weights to be nonnegative gives a cone, a different set from a span. Requiring a+b=1a+b=1 describes an affine combination, another different set. Counting two vectors without checking their directions can mistake a redundant pair for a plane-spanning one. Finally, a span always passes through the origin: it is a set of moves, not a choice of where to place a diagram.

Numerical considerations

The stage forms combinations and tests collinearity exactly from the accepted decimals. With approximate measurements, nearly collinear inputs may be hard to distinguish from exact dependence. A numerical rank estimate needs a scale-aware criterion; a fixed determinant threshold confuses small lengths with small directional separation. Large weights of opposite signs can also nearly cancel, making a small output sensitive to coefficient error. The exact stage makes the intended algebra inspectable, while the NumPy code follows finite-precision array arithmetic.

Engineering and ML connections

Combining actuator directions determines which displacements a mechanism can produce. Mixing feature directions determines which outputs a linear model can express. Redundant directions can add parameters without adding reachable outputs. Before asking how to fit weights, it helps to ask whether the desired result lies in the available span at all.

Exercises

calculation

Read the weights

u = (2, 1), v = (1, 2), a = 1, b = −0.5. Give w.

calculation

Two weights, one move

u = (1, 2), v = (2, 4). Give w for a = 3, b = −1.

construction

Reach a target

u = (1, 0), v = (1, 1). Choose a and b to reach (2, 3). Coefficients are unrestricted.

prediction

Predict beyond the control window

Two independent directions are shown with coefficients limited to [−2, 2]. Is their mathematical span only the shaded parallelogram?

Choose one answer

explanation

Both vectors become zero

Which points can au + bv reach when u = v = 0?

Choose one answer

Summary and checkpoints

Linear combinations scale and add available directions; span includes every result with unrestricted real weights. Check that you can distinguish a control window from a span, explain each collapse, and prove closure under addition and scaling. II.7 Basis and coordinates asks when the weights are unique.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available