Handling Missing Data, Part 1: Mean, Median, Mode & Flags

Handling missing data in machine learning: mean, median and mode imputation, dropping rows and missing-value flags, with scikit-learn's SimpleImputer. A small table of eight people has five empty cells. Whatever you do with them, dropping the rows or filling the cells, you change the data. This video shows the simple repairs one by one and measures what each of them costs, on numbers small enough to check by hand. You will learn: Three reasons a value goes missing, in plain words: pure chance, explained by other columns, or caused by the hidden value itself Complete case analysis (dropping rows): five holes cost five of the eight rows Filling a number with the mean: the mean stays, the variance drops from 200 to 150 Mean or median when there is an outlier Categories: the most frequent value, or a new "Missing" category Random-sample filling: it keeps the spread, but adds chance The missing indicator column, and why "it was missing" can itself be information Why the filling value is computed from the training rows only (SimpleImputer in scikit-learn) Part 2 covers the methods that use the other columns. Chapters 0:00 A table with holes 0:39 Three reasons a value goes missing 1:58 Dropping rows, and what it costs 3:05 Filling with the mean 3:23 What it does to the variance 4:05 Mean or median? 4:26 Categories: most frequent, or "Missing" 4:56 Random-sample filling 5:24 The missing indicator 5:59 Fit on the training data only 6:30 Recap + try it yourself Try it yourself: a column has the values 2, 4, 6, 8 and one hole. Fill the hole with the mean, then compare the variance before and after (divide by the number of values). Animations made with Manim Community Edition. The narration is a synthetic voice. #MachineLearning #MissingData #AIEngineer More from Harban AI Labs: https://harban-ai-labs.vercel.app/