From BERT to UltraBERT: 8 Years of Encoder Models in One Video

From BERT to UltraBERT: 8 Years of Encoder Models in One Video In 2018 BERT changed NLP. Eight years and a dozen landmark papers later, UltraBERT (arXiv 2610.15028) is the first encoder to be state of the art at BOTH classification (91.0 GLUE) and retrieval (47.7 BEIR). This video walks the whole family tree from the papers themselves: BERT's masked language modeling, RoBERTa's “BERT was undertrained”, ALBERT and DistilBERT, ELECTRA's replaced token detection, DeBERTa's disentangled attention, the “Should you mask 15%?” debate, MosaicBERT, Nomic, ModernBERT, NeoBERT and Ettin — and then UltraBERT's three findings: encoders saturate after 1–2T tokens, low masking helps classification while 50% masking helps retrieval, and distillation only helps at the low ratio. The fix: mix 5% and 50% masking in every batch with a separate LM head for each, and distill only the 5% half. — CHAPTERS — 0:00 Best at both, for the first time 0:52 Encoders vs decoders, and their two jobs 2:07 BERT (2018): bidirectional + masked LM 4:22 RoBERTa (2019): BERT was undertrained 5:27 ALBERT & DistilBERT: going smaller 6:14 ELECTRA (2020): spot the fake token 7:14 DeBERTa: disentangled attention 7:53 Should you mask 15%? 8:30 MosaicBERT & Nomic: the modern wave 9:10 ModernBERT: an LLM-style encoder 10:23 NeoBERT: one change at a time 10:55 Ettin: encoder vs decoder 11:39 Still no all-rounder 12:20 UltraBERT: a new recipe 12:53 Finding 1: encoders saturate 13:45 Finding 2: the masking ratio 16:15 The fix: decoupled LM heads 17:34 Finding 3: distill only the low ratio 18:08 The UltraBERT recipe 19:07 Results: the family tree in one chart 20:55 32K long context 21:23 Caveats & the 8-year arc — PAPERS (arXiv) — BERT: https://arxiv.org/abs/1810.04805 RoBERTa: https://arxiv.org/abs/1907.11692 ALBERT: https://arxiv.org/abs/1909.11942 DistilBERT: https://arxiv.org/abs/1910.01108 ELECTRA: https://arxiv.org/abs/2003.10555 DeBERTa: https://arxiv.org/abs/2006.03654 Should You Mask 15% in MLM?: https://arxiv.org/abs/2202.08005 MosaicBERT: https://arxiv.org/abs/2312.17482 Nomic Embed: https://arxiv.org/abs/2402.01613 ModernBERT: https://arxiv.org/abs/2412.13663 NeoBERT: https://arxiv.org/abs/2502.19587 Ettin (Seq vs Seq): https://arxiv.org/abs/2507.11412 UltraBERT: https://arxiv.org/abs/2610.15028 Figures are cropped from the papers and credited on screen; the animated charts are redrawn from the papers' own tables. Numbers were checked against each paper. More explainers: https://sumityadav.com.np #BERT #NLP #MachineLearning #ModernBERT #UltraBERT #Transformers #AIResearch