Linear Regression Explained - Basis Functions, Least Squares and the Bias-Variance Tradeoff

Twenty curves, each fitted to its own test drives, and every one is badly wrong. Average a hundred of them and you get almost exactly the true curve. Here is the maths of why, and of linear regression itself. The simplest neural network, a single layer of weights on fixed basis functions, built on one car's fuel use against speed and proved on screen: tanh and Gaussian basis functions, maximum likelihood and the pseudo-inverse, why the bias weight matches the averages, the geometry of least squares as an orthogonal projection, sequential learning with the LMS update (and why a learning rate that is too large explodes), weight decay versus the lasso and why the lasso sets weights exactly to zero, multiple outputs, decision theory (the conditional mean, median and mode as best guesses) and the bias-variance decomposition. Chapters 0:00 Why averaging 100 bad fits gives the right curve 0:19 Basis functions and the single-layer network 1:40 Maximum likelihood, the pseudo-inverse and the bias weight 2:44 The geometry of least squares: an orthogonal projection 4:02 Sequential learning: the LMS update and the learning-rate limit 5:04 Weight decay vs the lasso: why weights hit exactly zero 6:16 Multiple outputs share one pseudo-inverse 6:44 Decision theory: mean, median and mode 8:06 The bias-variance decomposition 9:30 Recap and three questions for you Questions to think about (answer in the comments): 1. If every fuel reading went up by exactly 1 L/100 km, which weights would change, and by how much? 2. If the cold-engine trips used 1 L more, how far would the best guess under squared loss move? 3. If every data set had 1000 drives instead of 25, which of bias, variance and noise could never shrink? Subtitles: English, Hindi. Part of the playlist "Deep Learning Basics". Further reading: C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts (Springer, 2024), chapter 4. This video contains AI-generated content: the narration is AI-synthesised and the visuals are generated programmatically. © 2026 The Deep Learn. All rights reserved. #DeepLearning #MachineLearning #LinearRegression