Tokens, Training, Attention: The Engine Behind Every AI Agent

Every AI agent has a part inside it doing the actual thinking — and it has no idea what words are. It reads tokens, and writes one token at a time. This is the engine episode. Last time we met the language model as the brain of an agent. Today we open the engine itself. First the smallest unit of everything: the token, and the autoregressive loop — write one piece, feed it back, predict the next — which explains why everything in your context window is counted and billed in tokens. Then the three formats that turn raw text prediction into behavior: the system prompt that outranks the user, the turns of a conversation, and tool calls — the structured text where the model doesn't act, it asks, and software around it does the acting. Then the endpoint contract — one OpenAI-compatible interface for a laptop model or a data center — and the trajectory, the harness's receipt for every run. Then the part almost nobody shows: how the model got its abilities. Pre-training's predict-compare-nudge loop at planetary scale, and the strange result — a base model that knows everything about language and cannot follow one instruction. Post-training's two steps: supervised fine-tuning, then preference learning. And RLVR — reinforcement learning with verifiable rewards — the trick that made models reliable at following formats, with a referee that can't be fooled. And then the deep dive, from the inside out. The transformer stack of 2020 and its two-part block. One forward pass, start to finish. Self-attention — the most important single operation in modern AI — walked through Q, K, and V with an actual calculation. Why attention recomputes, and what the KV-cache stores instead. Then the three inventions that made attention radically cheaper: multi-head latent attention, sparse attention (the newest step, from DeepSeek), and mixture of experts — the architecture hiding in plain sight in model names, where each token is routed to a few small experts instead of one giant one. Same engine, radically cheaper. Every diagram is hand-drawn and animated from scratch — a full motion-graphics explainer, no stock footage, no slide decks. Chapters: 0:00 Cold open — the engine inside every agent 1:10 Tokens — how a model really reads and writes 2:27 System prompts, turns, tool calls — the three formats 3:46 The endpoint contract — one API for any model 5:19 Trajectories — the harness keeps a receipt 6:22 Pre-training — predict, compare, nudge 7:31 Post-training — teaching the model to behave 9:02 RLVR — reinforcement learning with verifiable rewards 10:35 The transformer — the dominant design 12:06 One forward pass, start to finish 13:35 Inside the block — two parts 14:54 Self-attention — the core operation 16:10 Q, K, V — how attention computes 17:28 The KV-cache — attention is not free 18:43 MLA — compressing the cache 20:14 Sparse attention — the newest step 21:23 Mixture of experts — hiding in plain sight 23:29 The map, one pass 25:05 Your move If this made the engine click, subscribe — deep dives on the tools behind the terminal, twice a month.