Muon vs AdamW, Lion and Sophia: a fair race | Minimum Viable GPT #23

All our training scripts use AdamW, so in this episode we put it in a fair race with Lion, Sophia and Muon, each written from its paper, with tests. They train one model on one set of batches for a thousand steps, each at its best learning rate from a sweep, with three seeds. Muon reaches in 1,000 steps the loss AdamW needs about 1,840 for, while Lion and Sophia land behind a tuned AdamW. The series: https://mvgpt.nenuadrian.com The code: https://github.com/nenuadrian/mvGPT Run it in Colab: https://colab.research.google.com/git... The playlist:    • Minimum Viable Generative Pre-trained Tran...   Previous episode:    • Long context LLM: stretching RoPE with YaR...   Chapters: 0:00 Intro 2:28 What an optimizer does 8:04 A fair race 17:04 Lion 21:39 Sophia 29:18 Muon 35:48 The results 41:59 What's real, and what's scaled down 44:29 Wrap-up To train on GPUs you rent by the hour, cheaply, I recommend Vast.ai (referral link): https://cloud.vast.ai/?ref_id=746308 Music: John Tramp (https://pixabay.com/users/tramp963-35...) #llm #pytorch #machinelearning