The Trick Behind Google's 1.6 Trillion Parameter Model

Google's Switch Transformer broke a long-standing assumption in mixture-of-experts models — that you need to route each token to at least two experts to get a useful training signal. Switch Transformers route to just one, and it turns out to be simpler, faster, and just as good. That one change is what let researchers train models with over a trillion parameters without a proportional blowup in compute. This video walks through the paper (Fedus, Zoph & Shazeer, 2021) end to end: why dense scaling hits a hard ceiling, why earlier mixture-of-experts models never caught on, the counterintuitive "switch routing" insight, how it's wired into the transformer block, the practical engineering (expert capacity, the load-balancing loss, precision/init/dropout fixes that keep training stable), the head-to-head results against dense and top-2 MoE baselines, how it scales to 1.6 trillion parameters, and how those gains get distilled back into a small deployable model. 00:00 – The scaling promise: why bigger models cost more, every time 00:50 – Why sparsity never quite worked before 1:40 – The standard approach: route to the top two experts 2:25 – The insight: route to just one 3:16 – Where switch routing lives in the transformer block 4:04 – Expert capacity and token overflow 4:55 – The auxiliary loss that keeps routing balanced 5:45 – Three fixes that keep training from blowing up 6:37 – Does it actually beat the alternatives? 7:19 – The speed win: 7x faster, same quality 8:09 – Scaling to 1.6 trillion parameters 9:06 – Shrinking it back down: distillation 9:55 – What it unlocks Paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity" — Fedus, Zoph, Shazeer (JMLR 2022), arXiv:2101.03961