Gradient Boosting, Part 2: Classification, Step by Step

Gradient boosting for classification explained: log-odds, the sigmoid and residuals, two full rounds by hand. A regression tree can only say numbers. So how can a team of such trees give a probability? In this video we run gradient boosting for classification by hand, for two full rounds, on six made-up students. The trees add up scores in log-odds, and the sigmoid turns the total into a probability. We compute the starting score, the residuals, a small tree on the residuals, the value of each leaf, and the update with the learning rate, and we check every number against scikit-learn. Then we watch the decision border of a real model after 1, 5, 20 and 100 trees, and see why more trees are not always better. You will learn: Why the model adds scores (log-odds) and not probabilities The starting score: the log of the odds of the positive class (log 2 = 0.69, a probability of 0.67) Residuals: label minus probability, which is the negative gradient of the log loss Why a leaf value is the sum of residuals divided by the sum of p × (1 − p): one Newton step The update with the learning rate: 0.67 becomes 0.31 for the fails and 0.81 for the passes, then 0.18 and 0.89 The same numbers from scikit-learn's GradientBoostingClassifier (n_estimators=2, max_depth=1, learning_rate=0.5) The border of a real model after 1, 5, 20 and 100 trees, and the training and test log loss More than two classes: one tree per class in every round Chapters 0:00 How can trees give a probability? 0:45 Scores in log-odds and the sigmoid 1:25 The data and the starting score 1:57 Residuals: label minus probability 2:30 A small tree on the residuals 3:02 The leaf value: one Newton step 3:52 The update with the learning rate 4:30 Round two 5:10 Check against scikit-learn 5:25 The border after 1, 5, 20, 100 trees 6:08 Training and test log loss 6:43 Recap + try it yourself Try it yourself: keep the six students, but suppose that three passed and three failed. What is the starting score? What is the starting probability? And what is the first residual of a student who passed? Animations made with Manim Community Edition. The narration is a synthetic voice. All data in this video are made up. #GradientBoosting #MachineLearning #Classification More from Harban AI Labs: https://harban-ai-labs.vercel.app/