Stretching an LLMs context window (YaRN)

Why does a language model fall apart past the length it was trained on, and how can a short fine-tune stretch it from 4K to 128K tokens? This lesson rebuilds YaRN (Peng, Quesnelle, Fan & Shippole) from rotary position embeddings up, following a passkey hidden in a 64K-token document. • How RoPE writes position: 64 pairs turning like clock hands at different speeds, so that attention sees distances. • Why models break past the window: slow pairs reach angles never seen in training. • Position Interpolation squeezes every pair and blurs neighbouring tokens. NTK-aware interpolation changes the base and overshoots. • NTK-by-parts counts each pair's turns inside the window: fast pairs untouched, slow pairs interpolated, a blend in between. • An attention temperature (√(1/t) = 0.1 ln s + 1), folded into the rotation tables at zero cost. Together these are YaRN. • Dynamic scaling at inference, and what the experiments show: 128K from 64K training chunks, passkey retrieval, efficiency, limits. All head quantities are computed for a Llama-2-shaped head (labelled on screen). Toy examples are constructed; reported numbers come from the paper. *Chapters* 00:00 Past the edge 01:13 How rotary embeddings write position 03:00 Why the model breaks 04:28 Squeezing positions 05:47 Treating pairs differently 09:03 A temperature for attention 11:09 Stretching on the fly 12:09 What the experiments show 14:08 The passkey, at sixty-four thousand *Sources* • Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole, "YaRN: Efficient Context Window Extension of Large Language Models", arXiv:2309.00071 — https://arxiv.org/abs/2309.00071 • Jianlin Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding", arXiv:2104.09864 — https://arxiv.org/abs/2104.09864 • Shouyuan Chen et al., "Extending Context Window of Large Language Models via Positional Interpolation", arXiv:2306.15595 — https://arxiv.org/abs/2306.15595 • Amirkeivan Mohtashami, Martin Jaggi, "Landmark Attention: Random-Access Infinite Context Length for Transformers" (passkey retrieval), arXiv:2305.16300 — https://arxiv.org/abs/2305.16300 *Hashtags* #YaRN #RoPE #LLM #LongContext #ContextWindow #PositionalEncoding #MachineLearning #DeepLearning #Transformers #AIResearch