How AI Language Models Work: Tokens, Attention and Next-Word Guessing

Every AI chatbot runs on one core trick: look at the text so far and predict what comes next. Here's how that simple game, played at enormous scale, turns into something that can write, explain and code. Tokens, training as a guessing game, embeddings as a map of meaning, attention and the transformer, temperature, how assistants are fine-tuned with human feedback, context windows, why models confidently make things up, and what they're good and bad at. CHAPTERS 0:00 The cat sat on the… 0:49 Tokens 1:37 Training: the guessing game 2:50 Embeddings: a map of meaning 3:41 Attention and the transformer 4:59 One token at a time: temperature 5:51 From model to assistant 6:54 Context windows: working memory 7:36 Why AI makes things up 8:42 Good at, bad at 9:27 Back to the cat SOURCES • Vaswani et al., “Attention Is All You Need” (arXiv, June 2017) • C. E. Shannon, “Prediction and Entropy of Printed English,” Bell System Technical Journal (1951) • Mikolov, Yih & Zweig, “Linguistic Regularities in Continuous Space Word Representations” (2013) • Ouyang et al., “Training language models to follow instructions with human feedback” (2022) • OpenAI, “Why Language Models Hallucinate” (2025) · OpenAI Help Center, “What are tokens?” • Mata v. Avianca sanctions order, S.D.N.Y. (June 22, 2023), via LawNext and ABC News • Ted Chiang, “ChatGPT Is a Blurry JPEG of the Web,” The New Yorker (Feb 2023) Here by Degrees: how things got this way, one step at a time. New explainers every week. Subscribe so you don't miss the next one. #AI #LLM #machinelearning