Gradient Descent From Scratch (Linear Regression, Visually)

How does a machine actually learn? It walks downhill, blindfolded. We take the delivery-time model from episode two and let it find its own knob settings: a loss that measures how wrong it is (mean squared error), the loss landscape, the gradient as the slope under your feet, and gradient descent stepping downhill. Then the learning rate (too small crawls, too big explodes), reading a loss curve, feature scaling, batch vs stochastic vs mini-batch, why we don't just solve it, and what one outlier does to your line. All with real numbers from code you can run. Episode 3 of "From a Straight Line to ChatGPT", a full series on AI, machine learning and deep learning for developers and interview prep. ⏱ Chapters 0:00 Intro 0:53 The setup 1:14 The loss: mean squared error 1:58 The loss landscape 2:27 Downhill, blindfolded 3:07 The one equation 4:03 Rolling to the bottom 4:35 The learning rate 5:25 Reading the loss curve 5:56 Make the valley round 6:26 Batch, stochastic, mini-batch 7:11 Why not just solve it? 7:48 What can go wrong: outliers 8:29 Try it yourself 9:08 Interview corner 10:11 Recap and next stop 🧪 Try it yourself (Python + NumPy) pip install numpy scikit-learn import numpy as np from sklearn.linear_model import LinearRegression The same 10 made-up deliveries as episode 2 (hidden rule: 8 minutes + 3 per km, plus noise) rng = np.random.default_rng(1) km = rng.uniform(1, 10, size=10).round(1) minutes = (8 + 3 * km + rng.normal(0, 3, size=10)).round() w, b = 0.0, 0.0 # the two knobs, starting anywhere lr = 0.01 # learning rate: how big a step to take for step in range(5001): err = w * km + b - minutes loss = np.mean(err ** 2) # mean squared error if step in (0, 1, 10, 100, 1000, 5000): print(f"step {step:4d} loss {loss:8.3f} w {w:6.3f} b {b:6.3f}") w -= lr * 2 * np.mean(err * km) # step downhill along w b -= lr * 2 * np.mean(err) # ...and along b print(f"gradient descent: minutes = {w:.2f} x km + {b:.2f}") check = LinearRegression().fit(km.reshape(-1, 1), minutes) print(f"scikit-learn: minutes = {check.coef_[0]:.2f} x km + {check.intercept_:.2f}") What we got: loss 661.4 at step 0, 37.7 after one step, 1.373 after 5,000, landing on 3.17 per km + 6.52, the same line as scikit-learn. Now try lr = 0.001 (it crawls) and lr = 0.03 (it blows up). 🧠 Cheat sheet Loss (mean squared error) = average of (prediction − actual)² Gradient = the slope of the loss for each knob; it points uphill, so step the other way Update: knob = knob − learning rate × slope Learning rate too small: crawls · too big: overshoots and diverges · here, anything above ~0.025 blows up Loss curve: drops then flattens = healthy · still falling = too slow · shoots up = diverging Rescale your features: here 822 steps became 25 Batch = all data per step · stochastic = one example · mini-batch = a small chunk (the deep-learning default) · epoch = one pass over the data Linear regression can be solved exactly; neural networks can't, so they're trained by gradient descent (often Adam) Squared error is sensitive to outliers; robust options include Huber loss 📚 Sources Legendre, least squares (1805), transcription: https://www.york.ac.uk/depts/maths/hi... Stigler, Gauss and the Invention of Least Squares (1981): https://doi.org/10.1214/aos/1176345451 Lemaréchal, Cauchy and the Gradient Method (2012): https://doi.org/10.4171/DMS/6/27 Robbins & Monro, A Stochastic Approximation Method (1951): https://doi.org/10.1214/aoms/1177729586 Kingma & Ba, Adam (2014): https://arxiv.org/abs/1412.6980 Brown et al., GPT-3 (2020): https://arxiv.org/abs/2005.14165 Huber, Robust Estimation of a Location Parameter (1964): https://doi.org/10.1214/aoms/1177703732 Goodfellow, Bengio & Courville, Deep Learning (2016): https://www.deeplearningbook.org/ 📚 Series: From a Straight Line to ChatGPT AI01 What Happens When You Ask ChatGPT a Question? AI02 What Is Machine Learning, Really? AI03 How Machines Learn: Linear Regression & Gradient Descent ← you are here AI04 Yes or No? Logistic Regression & Classification … full playlist:    • Artificial Intelligence (AI)   Just Known: from unknown to known. A channel by Just Unknown. Note: the delivery data is made up from a hidden rule (as episode 2 says); every number on screen is real output from the code above. The loss landscape is computed from that data; pictures marked "illustration" are not data. Narration voice generated with Google Gemini text-to-speech. #ai #machinelearning #gradientdescent #python #computerscience #codinginterview