Speculative Decoding Explained: Same Answer, Faster (Local LLMs)

Make the AI on your own computer faster without changing a single word it writes. That's speculative decoding, and this video shows why it works, how to turn it on, and when it doesn't help. Companion repo (toy, launch scripts for llama.cpp / vLLM / SGLang / MLX, and a benchmark that checks outputs are identical): https://github.com/sreejithsr441/Spec... In this video: why a model on your laptop is slow (it's the reading, not the maths), the sentence-finishing friend (guess and check), why checking five guesses costs about the same as one, why the answer is identical and not just similar, where the guesses come from (draft models, n-gram copying, MTP and EAGLE), and how to switch it on for a Mac, a single GPU or a CPU+GPU split. Chapters 0:00 Same answer, faster? 0:36 Why your local model is slow 1:59 Guess and check 3:09 Why checking is nearly free 5:28 Where the guesses come from 6:18 Turning it on: Mac, GPU, CPU+GPU split 7:53 When it doesn't help 8:44 The race, explained Try it: clone the repo, then pip install -r toy/requirements.txt python toy/speculative_toy.py No graphics card needed. Sources Leviathan, Kalman, Matias (2022), Fast Inference from Transformers via Speculative Decoding: https://arxiv.org/abs/2211.17192 Chen et al., DeepMind (2023), Accelerating Large Language Model Decoding with Speculative Sampling: https://arxiv.org/abs/2302.01318 llama.cpp speculative decoding docs (draft model, n-gram, ~16 MB): https://github.com/ggml-org/llama.cpp... vLLM speculative decoding docs: https://docs.vllm.ai/en/latest/featur... SGLang speculative decoding docs: https://docs.sglang.io/advanced_featu... mlx-lm server (draft_model, num_draft_tokens): https://github.com/ml-explore/mlx-lm/... Apple unified memory (M1 announcement): https://www.apple.com/newsroom/2020/1... What's your setup, and what speedup did the repo measure? Post your number in the comments. #SpeculativeDecoding #LocalLLM #LLM