Mathematical Foundations

II.11 · Linear Algebra

Transpose the frame; test whether geometry survives.

Exchange rows and columns, compute QᵀQ exactly, and compare rotations, reflections, shears, and scalings through the lengths and angles they preserve—or fail to preserve.

Transpose the frame; test what it preserves

The columns are an orthonormal frame. Q preserves dot products, so lengths and angles survive; a reflection may still reverse orientation.

matrix A
2×2 [0, −1; 1, 0]
transpose
2×2 [0, 1; −1, 0]
orthogonality test
2×2 [1, 0; 0, 1]
column dot product
0
input length
√52.236068…
output length
√52.236068…
angle between x and y
1.570796
angle between Qx and Qy
1.570796
inverse-by-transpose test
2×2 [0, 1; −1, 0]

Exact expressions use =. Rounded computations use ≈. Every view uses this same state.

The columns are an orthonormal frame. Q preserves dot products, so lengths and angles survive; a reflection may still reverse orientation.

QTQ=[1001]=IQ^\mathsf TQ=\begin{bmatrix}1&0\\0&1\end{bmatrix}=I
∥Qx∥=5,(Qx)⋅(Qy)=0\norm{Q\mathbf x}=\sqrt{5},\quad (Q\mathbf x)\cdot(Q\mathbf y)=0
Q−1=QTQ^{-1}=Q^\mathsf T

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

from −4 to 4, up to 2 decimal places

Drag the two columns of Q. The stage computes QᵀQ, compares input and output lengths and angles, and distinguishes rotations and reflections from shears and unequal scaling.

Start with Rotation and compare the two columns. Choose Reflection and notice that lengths and angles still survive even though orientation changes. Then choose Shear fails and move a column until the off-diagonal dot products vanish again.

Intuition: rows become columns

The transpose turns a matrix around its main diagonal. This simple exchange lets matrix multiplication collect column dot-products:

(QTQ)ij=qi⋅qj.(Q^{\mathsf T}Q)_{ij}=\mathbf q_i\cdot\mathbf q_j.

The diagonal entries are squared column lengths. The off-diagonal entries are pairwise dot products. One product therefore asks whether the columns form an orthonormal frame.

Definition

For an m×nm\times n matrix AA, its transpose ATA^{\mathsf T} is the n×mn\times m matrix defined by

(AT)ij=Aji.(A^{\mathsf T})_{ij}=A_{ji}.

A square matrix QQ is orthogonal when

QTQ=I.Q^{\mathsf T}Q=I.

Equivalently, its columns are unit vectors and mutually orthogonal. In two dimensions, these columns form a rigid coordinate frame.

Mechanics: a certificate in one product

For Q=[q1 q2]Q=[\mathbf q_1\ \mathbf q_2],

QTQ=[q1⋅q1q1⋅q2q2⋅q1q2⋅q2].Q^{\mathsf T}Q= \begin{bmatrix} \mathbf q_1\cdot\mathbf q_1&\mathbf q_1\cdot\mathbf q_2\\ \mathbf q_2\cdot\mathbf q_1&\mathbf q_2\cdot\mathbf q_2 \end{bmatrix}.

The identity result says both columns have length one and their dot product is zero. A shear fails through a nonzero off-diagonal entry. Unequal scaling fails through a diagonal entry other than one.

Derivation: dot products, lengths, and angles survive

If QTQ=IQ^{\mathsf T}Q=I, then for any x,y\mathbf x,\mathbf y,

(Qx)⋅(Qy)=(Qx)T(Qy)=xTQTQy=xTy.(Q\mathbf x)\cdot(Q\mathbf y) =(Q\mathbf x)^{\mathsf T}(Q\mathbf y) =\mathbf x^{\mathsf T}Q^{\mathsf T}Q\mathbf y =\mathbf x^{\mathsf T}\mathbf y.

Setting y=x\mathbf y=\mathbf x proves ∥Qx∥2=∥x∥2\|Q\mathbf x\|^2=\|\mathbf x\|^2. Since dot products and lengths determine the cosine formula, angles between nonzero vectors are preserved too.

The transpose as an inverse—only here

The equation QTQ=IQ^{\mathsf T}Q=I says that QTQ^{\mathsf T} undoes QQ. For a square orthogonal matrix,

Q−1=QT.Q^{-1}=Q^{\mathsf T}.

This is a special structural shortcut, not a rule for arbitrary matrices. A shear’s transpose does not undo the shear, and a scaling transpose is usually the same scaling rather than its reciprocal. General inverses remain for II.16.

Rotations and reflections

A planar rotation by angle θ\theta has columns (cos⁡θ,sin⁡θ)(\cos\theta,\sin\theta) and (−sin⁡θ,cos⁡θ)(-\sin\theta,\cos\theta). A reflection also has orthonormal columns but reverses handedness. The interactive presets use exact quarter-turn and axis-reflection entries so the certificate is exact rather than tolerance-based.

Worked examples

For

Q=[0−110],QT=[01−10],Q=\begin{bmatrix}0&-1\\1&0\end{bmatrix}, \qquad Q^{\mathsf T}=\begin{bmatrix}0&1\\-1&0\end{bmatrix},

direct multiplication gives QTQ=IQ^{\mathsf T}Q=I. The vector (3,4)(3,4) becomes (−4,3)(-4,3); both have length 55.

Inspect the worked calculation

Transpose Q by exchanging rows and columns.

QTQ=[1001]=IQ^\mathsf TQ=\begin{bmatrix}1&0\\0&1\end{bmatrix}=I
matrix A
2×2 [0, −1; 1, 0]
transpose
2×2 [0, 1; −1, 0]
orthogonality test
2×2 [1, 0; 0, 1]
column dot product
0
input length
√52.236068…
output length
√52.236068…
angle between x and y
1.570796
angle between Qx and Qy
1.570796
inverse-by-transpose test
2×2 [0, 1; −1, 0]
Exchange rows and columns, multiply QᵀQ, then read the geometric preservation certificate.

For the shear [1101]\begin{bmatrix}1&1\\0&1\end{bmatrix}, the columns have dot product 11, so QTQ≠IQ^{\mathsf T}Q\ne I. Some individual vectors may keep their length, but the map has no global preservation guarantee.

Executable reference code

The Python reference transposes and multiplies exact rational matrices, then asserts the identity certificate. The NumPy version compares dot products, lengths, and a reflection’s orientation while using an explicit tolerance for floating-point checks.

code/python/matrix_transpose_orthogonality.py
from fractions import Fraction as F


def transpose(a):
    return tuple(zip(*a))


def matmul(left, right):
    return tuple(tuple(sum((x * y for x, y in zip(row, column)), F(0)) for column in zip(*right)) for row in left)


Q = ((F(0), F(-1)), (F(1), F(0)))
I = ((F(1), F(0)), (F(0), F(1)))
assert matmul(transpose(Q), Q) == I
assert transpose(Q) == ((F(0), F(1)), (F(-1), F(0)))
code/python/matrix_transpose_orthogonality_numpy.py
import numpy as np

Q = np.array([[0.0, -1.0], [1.0, 0.0]])
x, y = np.array([2.0, 1.0]), np.array([-1.0, 2.0])

np.testing.assert_allclose(Q.T @ Q, np.eye(2))
np.testing.assert_allclose(np.linalg.inv(Q), Q.T)
np.testing.assert_allclose(np.linalg.norm(Q @ x), np.linalg.norm(x))
np.testing.assert_allclose((Q @ x) @ (Q @ y), x @ y)

Explore

Swap the two identity columns and identify the resulting reflection. Negate both columns and recognize a half-turn. Give one column length two and inspect the diagonal failure. Make unit-length columns that are not perpendicular, then perpendicular columns with unequal lengths. Explain each entry of QTQQ^{\mathsf T}Q.

Common mistakes

Transposition does not negate entries, and it does not reverse the order inside one vector. Perpendicular columns alone are not orthonormal unless each has unit length. Orthogonal matrices need not be symmetric, and reflections are not excluded. Most importantly, A−1=ATA^{-1}=A^{\mathsf T} is false without the orthogonality certificate.

Numerical considerations

Exact entered decimals give an exact QTQQ^{\mathsf T}Q in this lesson. Floating-point code typically tests whether it is close to II with a scale-aware tolerance. That numerical test is evidence about stored approximations, not a redefinition of orthogonality. Angle calculations should also guard zero vectors and clamp only tiny endpoint roundoff before applying arccos.

Engineering and ML connections

Orthogonal transforms preserve signal energy and avoid stretching some directions more than others. They appear in rigid motion, numerical factorization, initialization, and feature rotations. Their transpose is cheap to apply and stable as an inverse, but learned matrices are not automatically orthogonal unless a model or algorithm enforces that structure.

Exercises

calculation

Transpose a matrix

Give Qᵀ for Q = [[0, −1], [1, 0]] by rows.

prediction

Certify a frame

QᵀQ = I. What does that certify about Q’s columns?

Choose one answer

calculation

Preserve a length

Q is orthogonal and x = (3, 4). What is ‖Qx‖?

prediction

Rotation or reflection?

An orthogonal matrix has determinant −1. Which possibility fits?

Choose one answer

explanation

Use the transpose shortcut carefully

When may you conclude Q⁻¹ = Qᵀ from QᵀQ = I?

Choose one answer

Summary and checkpoints

The transpose exchanges rows and columns; QTQQ^{\mathsf T}Q contains all column dot-products. Check that you can recognize an orthonormal frame, prove preservation of dot products, lengths, and angles, include reflections, and restrict the transpose-inverse shortcut to square orthogonal matrices.

The map of the book

97 of 97 chapters written · eight Parts

demo Foundation demo Try the live vector example Open as a page

Part I

Mathematical Language

The notation and mental objects everything else uses.

9 of 9 chapters written

  1. I.1 Scalars, quantities, units, powers and roots available
  2. I.2 Coordinates and points available
  3. I.3 Vectors: components, magnitude, and direction reference chapter available
  4. I.4 Algebraic expressions, equations and inequalities available
  5. I.5 Functions as input → transformation → output available
  6. I.6 Essential function families available
  7. I.7 Sets, intervals, and relations available
  8. I.8 Indexed quantities and summation available
  9. I.9 The notation map for ML available

Part II

Linear Algebra

Matrices as transformations: the strongest foundation in the book.

19 of 19 chapters written

  1. II.1 Vector addition and subtraction available
  2. II.2 Scalar multiplication and vector scaling available
  3. II.3 Norms, magnitude, and normalization available
  4. II.4 Dot product, angle, and orthogonality available
  5. II.5 Projection and distance to a direction available
  6. II.6 Linear combinations and span available
  7. II.7 Linear independence, basis, and coordinates available
  8. II.8 Matrices as linear transformations available
  9. II.9 Matrix × vector available
  10. II.10 Matrix × matrix as composition available
  11. II.11 Transpose and orthogonality available
  12. II.12 Determinant: orientation and area/volume scaling available
  13. II.13 Linear systems as intersecting constraints available
  14. II.14 Gaussian elimination and row operations available
  15. II.15 Rank and invertibility available
  16. II.16 Inverse matrices available
  17. II.17 Eigenvalues and eigenvectors available
  18. II.18 Orthogonal bases and least-squares geometry available
  19. II.19 Singular Value Decomposition available

Part III

Calculus

Change, local approximation, and sensitivity made visible.

15 of 15 chapters written

  1. III.1 Limits through approach available
  2. III.2 Continuity and local behavior available
  3. III.3 Derivative as secant → tangent available
  4. III.4 Derivative as rate of change and local linear model available
  5. III.5 Derivative rules available
  6. III.6 Chain rule available
  7. III.7 Multivariable functions and partial derivatives available
  8. III.8 Gradient and directional derivatives available
  9. III.9 Jacobian available
  10. III.10 Computational graphs and automatic differentiation available
  11. III.11 Hessian and curvature available
  12. III.12 Integrals as accumulation available
  13. III.13 Fundamental derivative/integral connection available
  14. III.14 Numerical differentiation available
  15. III.15 Numerical integration available

Part IV

Probability

Uncertainty through sample spaces, evidence, and repetition.

16 of 16 chapters written

  1. IV.1 Random experiments, outcomes, and sample spaces available
  2. IV.2 Events and probability mass available
  3. IV.3 Counting and finite probability where needed available
  4. IV.4 Conditional probability as restricted sample space available
  5. IV.5 Independence available
  6. IV.6 Bayes' theorem as probability-mass update available
  7. IV.7 Random variables available
  8. IV.8 PMF, PDF, and CDF available
  9. IV.9 Bernoulli and Binomial distributions available
  10. IV.10 Uniform and Gaussian distributions available
  11. IV.11 Expectation available
  12. IV.12 Variance and standard deviation available
  13. IV.13 Joint, marginal, and conditional distributions available
  14. IV.14 Covariance and correlation available
  15. IV.15 Law of Large Numbers available
  16. IV.16 Central Limit Theorem available

Part V

Statistics

From observed data to estimates, intervals, and likelihood.

10 of 10 chapters written

  1. V.1 Data summaries: mean, median, mode available
  2. V.2 Spread: variance, standard deviation, robust interpretation available
  3. V.3 Population, sample, and sampling variability available
  4. V.4 Estimators and estimation available
  5. V.5 Confidence intervals through repeated sampling available
  6. V.6 Covariance matrices and multivariate spread available
  7. V.7 Regression fundamentals available
  8. V.8 Likelihood available
  9. V.9 Maximum Likelihood Estimation available
  10. V.10 Maximum A Posteriori estimation available

Part VI

Optimization

Learning as repeated parameter updates governed by an objective.

10 of 10 chapters written

  1. VI.1 Objective functions and optimization landscapes available
  2. VI.2 Local vs global optima and stationary points available
  3. VI.3 Convexity available
  4. VI.4 Gradient descent flagship chapter available
  5. VI.5 Learning rate and convergence behavior available
  6. VI.6 Momentum available
  7. VI.7 Stochastic and mini-batch gradient descent available
  8. VI.8 Adam intuition available
  9. VI.9 Constraints and projected/penalized intuition available
  10. VI.10 Numerical stopping and optimization diagnostics available

Part VII

Information Theory

Logarithms, entropy, and the loss behind token prediction.

8 of 8 chapters written

  1. VII.1 Logarithms and log probabilities in computation available
  2. VII.2 Self-information available
  3. VII.3 Entropy available
  4. VII.4 Cross entropy and negative log-likelihood available
  5. VII.5 KL divergence available
  6. VII.6 Logits and softmax available
  7. VII.7 Numerical stability: log-sum-exp and stable softmax available
  8. VII.8 From token logits to language-model loss available

Part VIII

Numerical Computing

Why equivalent expressions behave differently on real machines.

10 of 10 chapters written

  1. VIII.1 Floating-point representation available
  2. VIII.2 Rounding and representation error available
  3. VIII.3 Error propagation and conditioning available
  4. VIII.4 Catastrophic cancellation available
  5. VIII.5 Overflow and underflow available
  6. VIII.6 Stable algorithms available
  7. VIII.7 Numerical linear algebra realities available
  8. VIII.8 Sparse matrices and sparse structure available
  9. VIII.9 Vectorized computation available
  10. VIII.10 Reproducibility, random seeds, and nondeterminism available