Memory-Based Speculative Decoding, Explained in 3 Minutes (INLG 2026)

How can a large language model generate text faster and with less energy? This animation shows speculative decoding in action, with a twist: the draft model is not a neural network but Olifant, a memory-based model that predicts the next token from the previous four in under 0.3 ms on a CPU, with no GPU and no backpropagation. The verifier is Mistral-7B. Two settings are shown side by side against plain decoding, with clocks in real milliseconds: Without a KV cache, draft tokens are almost free to check. Olifant drafts 8 tokens at a time, reaches 38% acceptance on EU legal text, and delivers 2.43× the throughput at 2.51× lower energy per token. With a KV cache, the verifier itself is ~10× faster and every draft token costs real time. Here a neural draft model (TinyLlama) loses ground, while Olifant still gives 1.25× throughput and the lowest energy in the study: 93 J per 100 tokens. Speculative decoding is not dead in the age of fast verifiers, provided drafting is nearly free. Chapters 0:00 Introduction 0:40 Without a KV cache 2:26 With a KV cache Paper: "Towards Green Text Generation: Memory-Based Speculative Decoding", Antal van den Bosch, Utrecht University. To appear in the Proceedings of INLG 2026 (link will be added when published). Code: https://github.com/antalvdb/olifant-s... Made with Cheetah3D; narration synthesised with ElevenLabs.