Least Squares: The Best Answer When There Is None

Least squares, the pseudoinverse, the trace and the derivation of PCA, explained visually from zero. Linear algebra for machine learning: why computers rarely invert a matrix, what the best-fit line really is, and how the Moore-Penrose pseudoinverse picks the closest and the shortest answer. In 1929, Edwin Hubble put twenty-four galaxies on one chart, and no straight line passes through all of them. Which line is best? First, how computers really solve equations: they almost never compute the inverse, because rounding adds error at every extra step (measured here on 1,000 equations). Then measure a desk twice, get 152 and 150 cm, and draw the pair as one point: every possible answer lies on the diagonal, and the closest is the perpendicular drop, a shadow, which lands on the average. By Pythagoras it has the least sum of squared errors: the method of least squares (Legendre, 1805). The same shadow gives Hubble's line, and with a starting height, a shadow on a plane. The pseudoinverse, built from the SVD by undoing what can be undone and leaving the rest at zero, finds the closest answer when there is none and the shortest when there are many (E. H. Moore, 1920; Roger Penrose, 1955). The same move compresses data: a point's best code is its shadows on PCA's directions, c = Dᵀx. With the trace and the Frobenius norm, the loss over all the data is the total minus what is kept, so losing the least is keeping the most: the top eigenvectors of XᵀX, found with the SVD. Part 5 of How Machines Learn, a series that builds the ideas behind modern AI from zero, one idea per episode. Next: probability, the mathematics of chance. For teachers: 1. Three measurements of one length give 10, 12 and 17. Show that the least-squares answer is their average. Why is the leftover error arrow perpendicular to (1, 1, 1)? 2. The table [½ ½] averages two numbers. What does its pseudoinverse do to the average 0.4, and why is that the shortest pair that averages 0.4? 3. Pick A (2 × 3) and B (3 × 2) and check that AB and BA have the same trace. Why must they, though the products have different shapes? Chapters 0:00 Twenty-four galaxies 0:58 Solve, don't invert 2:47 The closest point 4:35 Hubble's line 6:33 The best possible undo 8:54 Too many answers 10:43 The best code 13:18 Adding up a diagonal 14:47 Least loss, most spread 17:05 Why squares? #Latent #LinearAlgebra #MachineLearning