Stacking and Blending: A Model That Learns How to Combine Models

Stacking and blending in machine learning: a meta-learner that combines models, and the data leakage trap. Voting gives every model the same say. Stacking trains one more model, the meta-learner, on the predictions of the others, so that it learns whom to trust. This video shows the idea, the one trap that ruins it (leakage, with real numbers), the cure (out-of-fold predictions), the simpler cousin (blending), and a real StackingClassifier with an honest scoreboard. You will learn: The idea: base models on level one, a meta-learner on level two, trained on a table of predictions The trap: if the base models predict the rows they were trained on, the meta-learner trusts an overfitted model A real demonstration: leaky stacking 79.0 % against proper stacking 86.6 % on the same test data Out-of-fold predictions: the K-fold table, filled column by column Why the meta-learner is usually simple, and why the base models are fitted again on all the data Blending: one hold-out set instead of folds, and what it costs A real StackingClassifier: every base model, soft voting, hard voting and the stack, reported honestly The parameters: estimators, final_estimator, cv, stack_method, passthrough When stacking is worth the complexity, and when it is not All data in this video are made up for teaching. Chapters 0:00 A model that learns whom to trust 0:35 The idea: base models and a meta-learner 1:25 The obvious way, on a made-up table 1:59 A question: whom will it trust? 2:13 The trap: leakage, with real numbers 2:55 The cure: out-of-fold predictions 3:44 Training the meta-learner, then refitting 4:42 Blending: one hold-out set 5:26 StackingClassifier: the honest scoreboard 6:12 Parameters, and when stacking is worth it 7:02 Recap + try it yourself Try it yourself: you stack three base models on 1,000 training rows with 5 folds. (1) When a base model predicts one fold, on how many rows was it trained? (2) How many times is each base model trained in total, including the final training? Animations made with Manim Community Edition. The narration is a synthetic voice. #MachineLearning #Stacking #AIEngineer More from Harban AI Labs: https://harban-ai-labs.vercel.app/