Long context LLM: stretching RoPE with YaRN | Minimum Viable GPT #22

Our web model was pretrained on windows of 1,024 tokens, and its loss climbs past them, so in this episode we stretch its rotary embeddings four ways, from position interpolation to YaRN. We measure each on long web pages, by position, against a sliding window, and with the pass key test, a number hidden in filler. YaRN keeps the loss flat out to 4,096 tokens with no training at all, and then we fine-tune on long pages for fifteen minutes with two of the methods. The series: https://mvgpt.nenuadrian.com The code: https://github.com/nenuadrian/mvGPT Run it in Colab: https://colab.research.google.com/git... The playlist:    • Minimum Viable Generative Pre-trained Tran...   Previous episode:    • AI agent from scratch, with subagents | Mi...   Next episode:    • Muon vs AdamW, Lion and Sophia: a fair rac...   Chapters: 0:00 Intro 2:28 Past the window 8:17 Position interpolation 10:10 NTK-aware, and NTK-by-parts 12:56 YaRN's temperature 14:13 Stretching in code 22:52 The pass key 27:37 Fifteen minutes at 4,096 34:25 What a longer window costs 36:02 What's real, and what's scaled down 37:47 Wrap-up To train on GPUs you rent by the hour, cheaply, I recommend Vast.ai (referral link): https://cloud.vast.ai/?ref_id=746308 Music: John Tramp (https://pixabay.com/users/tramp963-35...) #llm #pytorch #machinelearning