Mathematical Foundations

II.17 · Linear Algebra

Which input directions survive a transformation on their original line?

Drag a nonzero vector and compare it with its image. Discover stretching, reversal and collapse, then contrast a scalar map, a defective shear and a rotation with no real eigenvector.

Discover the directions that stay on their line

Find an input line that survives A

Two distinct real eigenvalues give two independent eigen-directions. This chosen input leaves its original line.

Real eigen-directions
distinct
Trace
1
Exact characteristic discriminant
9
Chosen nonzero eigenvector gain
undefinedAv is not a scalar multiple of v.
Eigenvalue
2
Numerical eigenpair residual
0
Eigenvalue
−1
Numerical eigenpair residual
0

Exact values use =; numerical companions use ≈. Rank and zero decisions are exact.

Two distinct real eigenvalues give two independent eigen-directions. This chosen input leaves its original line.

A=[200−1],Av=[2−1]A=\begin{bmatrix}2&0\\0&-1\end{bmatrix},\quad A\mathbf v=\begin{bmatrix}2\\-1\end{bmatrix}
det⁡(A−λI)=λ2−(1)λ+(−2)=0\det(A-\lambda I)=\lambda^2-(1)\lambda+(-2)=0
λ±=0.5±1.5\lambda_{\pm}=0.5\pm 1.5
A[30]=(2)[30]A\begin{bmatrix}3\\0\end{bmatrix}=(2)\begin{bmatrix}3\\0\end{bmatrix}
A[0−3]=(−1)[0−3]A\begin{bmatrix}0\\-3\end{bmatrix}=(-1)\begin{bmatrix}0\\-3\end{bmatrix}

Derive the characteristic equation

Subtract λ on the diagonal and expand the determinant. The discriminant classifies real roots exactly. A = [2, 0; 0, −1].

det⁡(A−λI)=λ2−(1)λ+(−2)=0\det(A-\lambda I)=\lambda^2-(1)\lambda+(-2)=0

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

Manipulate the input and inspect the geometry, exact classification and replayable mathematical derivation.

Try aligning the input with each dashed eigen-line. In Stretch and reversal, compare the positive and negative gains. Then choose Quarter-turn: no real direction. Every nonzero input turns away from its original line; the absence of a real eigenvector is a mathematical result, not an unfinished drawing.

Intuition: special directions simplify repeated action

Most vectors change both length and direction under a matrix. Some remain on the same line through the origin. Along such a direction, the full matrix acts like one scalar: stretch, reverse or collapse. Repeating the map multiplies that scalar repeatedly, which can explain growth and decay without multiplying the full matrix at every step.

Staying on the same line allows reversal; it does not require pointing the same way. A negative gain is therefore as meaningful as a positive one.

Definition

For square AA, an eigenvector is a nonzero vector vv such that

Av=λv.Av=\lambda v.

The scalar λ\lambda is its eigenvalue. Positive λ\lambda preserves its orientation, negative λ\lambda reverses it, and λ=0\lambda=0 collapses the direction into the origin. The zero vector satisfies the equation for every scalar but is excluded: it identifies no direction.

All vectors associated with one eigenvalue, together with zero, form its eigenspace ker⁡(A−λI)\ker(A-\lambda I). A basis of that space counts independent eigen-directions; scaling an eigenvector does not create a new independent direction.

Mechanics: test a vector, then inspect the spectrum

The entered vector and its image are calculated exactly from the same rational matrix. Exact collinearity certifies whether it is an eigenvector; when it is, the inspector derives the exact scalar gain. The dashed spectral lines use checked numerical unit-vector companions. They are approximate drawings of the eigenspaces, not proofs from pixels.

Edit all four matrix entries or the input components, drag the vector, or use arrow keys. The live trace derives the characteristic equation and replays four exact applications of the current map. The worked trace below retains the canonical example. Numeric controls accept two decimal places in [−4,4][-4,4] and reject invalid input.

Derivation: why the characteristic equation appears

Rearrange the definition as (A−λI)v=0(A-\lambda I)v=0. A nonzero null vector exists exactly when A−λIA-\lambda I is singular. Thus

det⁡(A−λI)=0.\det(A-\lambda I)=0.

For A=[abcd]A=\begin{bmatrix}a&b\\c&d\end{bmatrix} this expands to

(a−λ)(d−λ)−bc=λ2−(a+d)λ+(ad−bc)=0.(a-\lambda)(d-\lambda)-bc =\lambda^2-(a+d)\lambda+(ad-bc)=0.

Write t=tr⁡At=\operatorname{tr}A, δ=det⁡A\delta=\det A and D=t2−4δD=t^2-4\delta. When D≥0D\ge0 the roots are (t±D)/2(t\pm\sqrt D)/2. Exact rational arithmetic decides the sign of DD; the existing radical representation preserves simple exact roots. A numerical approximation is labeled ≈\approx, and its eigenpair residual is checked separately.

Worked examples

For A=diag⁡(2,−1)A=\operatorname{diag}(2,-1), the characteristic polynomial is λ2−λ−2=(λ−2)(λ+1)\lambda^2-\lambda-2=(\lambda-2)(\lambda+1). The x-axis stretches by 22, while the y-axis reverses by −1-1. Starting with (1,1)(1,1), repeated applications give (2,−1)(2,-1), (4,1)(4,1), (8,−1)(8,-1) and (16,1)(16,1). The x-component grows; the y-component alternates.

Replay the worked derivation

Derive the characteristic equation

Subtract λ on the diagonal and expand the determinant. The discriminant classifies real roots exactly. A = [2, 0; 0, −1].

det⁡(A−λI)=λ2−(1)λ+(−2)=0\det(A-\lambda I)=\lambda^2-(1)\lambda+(-2)=0
Derive the characteristic equation and replay the exact repeated-vector states.

For A=[0210]A=\begin{bmatrix}0&2\\1&0\end{bmatrix}, the roots are ±2\pm\sqrt2. The positive eigen-direction satisfies y=x/2y=x/\sqrt2. The lesson retains the exact radical eigenvalues and shows approximate normalized directions with residuals; it does not invent rational eigenvectors for irrational directions.

Repeated roots: scalar versus defective

The matrix 2I2I has eigenvalue 22 repeated twice, and every nonzero input is an eigenvector. Any two independent directions can serve as an eigenbasis; the displayed axes are one convenient choice.

By contrast, A=[1101]A=\begin{bmatrix}1&1\\0&1\end{bmatrix} also has a repeated root, but (A−I)v=(v2,0)=0(A-I)v=(v_2,0)=0 forces v2=0v_2=0. Only one independent direction exists. This is a defective matrix. Its shear behavior cannot be explained by two independent real eigenvectors; repeated root count is not eigenspace dimension.

No real eigenvector

For the quarter-turn Q=[0−110]Q=\begin{bmatrix}0&-1\\1&0\end{bmatrix}, the characteristic equation is λ2+1=0\lambda^2+1=0. No real scalar satisfies it. Geometrically, every nonzero vector rotates through a right angle and leaves its original line. Over complex numbers the eigenvalues are ±i\pm i. This explains the boundary of the real picture; complex eigenvectors and complex geometry are outside this chapter.

The zero-eigenvalue preset instead has a real nonzero null direction. Collapse is allowed as an eigenvalue behavior; it is the zero input vector that is excluded.

Executable reference code

The pedagogical reference derives exact characteristic coefficients and high-precision real roots. NumPy computes eigenpairs and checks each residual, including a rotation whose spectrum is complex.

code/python/eigenvalues.py
"""Read real 2×2 cases from the characteristic equation, with exact rational coefficients."""
from fractions import Fraction as F
from decimal import Decimal, localcontext

def eigenvalues(a):
    p,q,r,s = [F(str(x)) for row in a for x in row]
    trace, determinant = p+s, p*s-q*r
    discriminant = trace*trace-4*determinant
    if discriminant < 0:
        return 'no real eigenvectors', []
    with localcontext() as ctx:
        ctx.prec = 50
        dec = lambda x: Decimal(x.numerator)/Decimal(x.denominator)
        h = dec(discriminant).sqrt()
        values = [(dec(trace)+h)/2,(dec(trace)-h)/2]
    scalar = q == r == 0 and p == s
    kind = 'every nonzero direction' if scalar else 'one direction' if discriminant == 0 else 'two directions'
    return kind, values

assert eigenvalues([[2,0],[0,-1]]) == ('two directions',[Decimal(2),Decimal(-1)])
assert eigenvalues([[1,1],[0,1]])[0] == 'one direction'
assert eigenvalues([[2,0],[0,2]])[0] == 'every nonzero direction'
assert eigenvalues([[0,-1],[1,0]])[1] == []
code/python/eigenvalues_numpy.py
import numpy as np
A = np.array([[2.,0.],[0.,-1.]])
values, vectors = np.linalg.eig(A)  # Eigenvectors are columns; they must be nonzero.
for value, vector in zip(values, vectors.T):
    assert np.linalg.norm(A @ vector-value*vector) < 1e-12
rotation = np.array([[0.,-1.],[1.,0.]])
assert np.all(np.abs(np.linalg.eigvals(rotation).imag) > 0)
# A defective matrix has one independent eigen-direction, despite two repeated roots.

Explore

Move the input onto and away from an eigen-line. Reverse it and verify the gain stays the same. Compare the two repeated-root presets. Set the input to zero and read why its gain is undefined. Follow repeated actions for a negative gain, a zero gain and a quarter-turn.

Common mistakes

Eigenvectors need not be unit vectors, and eigenvalues need not be positive. A repeated eigenvalue does not guarantee two independent eigenvectors. A matrix with real entries may have no real eigenvector. The determinant condition finds candidate eigenvalues; finding independent null vectors still determines the eigenspaces.

Numerical considerations

The exact discriminant class is independent of display precision. Numerical companions use binary64 and a residual bound of 256ϵmax⁡(1,∥A∥F)256\epsilon\max(1,\|A\|_F) for unit directions, where ϵ\epsilon is machine epsilon. This conservative rounding envelope is validated independently against Decimal roots and NumPy eigenpairs; it never classifies an exact zero.

The cancellation-prone quadratic root is recovered using λ1λ2=det⁡A\lambda_1\lambda_2=\det A. Near defective matrices can have highly sensitive eigendirections even with small residuals. If distinct numerical roots cannot be separated or the residual check fails, the lesson reports a numerical limitation instead of fabricating a direction.

Engineering and ML connections

Eigen-directions describe repeated linear dynamics and stability, and symmetric eigenproblems underlie covariance analysis and PCA. General matrices can have complex or defective spectra, so a real eigenbasis is not universal. SVD will provide an orthogonal input/output decomposition for every real matrix, using distinct left and right directions.

Exercises

calculation

Read a reversed direction

A = diag(2, −1), v = (0, 3). Find λ in Av = λv.

calculation

Derive the characteristic polynomial

A = [[2, 0], [0, −1]]. Give t and d in λ² − tλ + d.

prediction

Count eigendirections

A = [[1, 1], [0, 1]] has repeated eigenvalue 1. How many independent eigendirections?

Choose one answer

prediction

A quarter-turn counterexample

Does [[0, −1], [1, 0]] have a real eigenvector?

Choose one answer

prediction

A zero eigenvalue

Av = 0 for a nonzero v. What follows?

Choose one answer

Summary and checkpoints

Test Av=λvAv=\lambda v with v≠0v\ne0, derive the characteristic polynomial, distinguish eigenvalue multiplicity from eigenspace dimension, and explain reversal, collapse, repeated action and absence of real eigenvectors.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available