How 784 Numbers Become 20: Eigenvectors and PCA

Eigenvectors, eigenvalues, the singular value decomposition (SVD) and principal component analysis (PCA), explained visually from zero. Linear algebra for machine learning: why a handwritten digit's 784 numbers can be squeezed into twenty and still be read. Some arrows a matrix never turns: it only stretches them. These eigenvectors, with their stretch factors, the eigenvalues, show what a matrix really does, whatever axes it happens to be written in. Described along them, a move is just a stretch of each axis (the eigendecomposition A = V diag(λ) V⁻¹); its determinant is the product of its eigenvalues; and a zero eigenvalue means it flattens space and can never be undone. Not every matrix has enough of them: a turn has none and a slant only one. But every matrix, of any shape, is a turn, a stretch and another turn: the singular value decomposition A = U D Vᵀ. Then data. The pixel pairs of Hokusai's Great Wave crowd along a single direction, and 5,620 real handwritten digits spread along only a few of their 784. Principal component analysis finds the directions of greatest spread one at a time, each the longest of what the previous ones leave. Each direction is itself a picture, and the average plus twenty of them, each times its amount, rebuilds a 3 by a writer the computer never saw. Part 3 of How Machines Learn, a series that builds the ideas behind modern AI from zero, one idea per episode. Next: the dot product, and why the directions a cloud spreads along are eigenvectors. Chapters 0:00 Twenty numbers 0:59 What a matrix hides 2:20 Arrows that only stretch 3:45 The move in its own directions 5:57 What the stretches reveal 7:11 Turn, stretch, turn 9:37 Squeezing a picture 11:19 The directions that matter 14:39 A new description #Latent #LinearAlgebra #MachineLearning