How a Chatbot Writes One Word at a Time

A chatbot's answer flows onto your screen piece by piece, and that matches how it is made: one token at a time. This video follows that process step by step. How text is split into tokens. How a language model turns everything written so far into probabilities for every possible next token. Where those probabilities come from. How one token gets picked, and one reason the same question can get different answers. Mid-video, we train our own tiny next-word model on 18 classic books from Project Gutenberg (the selection included with the NLTK toolkit, 2,118,937 words) and show its real predictions. Our model is far smaller and simpler than the language models behind today's chatbots: 2,239,112 parameters, a 5,000-slot vocabulary (4,999 common words and punctuation marks plus one rare-words bucket), only the last four as context, and no attention. When it writes, it never picks the rare-words bucket. It was trained in 18.2 minutes on a four-core cloud computer with no GPU. What's real and what's illustrated: "REAL OUTPUT" and "REAL ODDS" mark numbers and text produced by our own model. "REAL COUNT" marks counts taken from our 18 training books. "ILLUSTRATION", "SIMPLIFIED" and "EXAMPLE VALUES" mark simplified visuals and made-up example numbers. "SIMULATION" marks simulated spins; "NOT TO SCALE" marks size comparisons that are not drawn to scale. CHAPTERS 0:00 Watch the answer arrive 0:38 What counts as a word? 1:19 What is a guess? 2:12 Where do the odds come from? 3:31 The experiment: our tiny model 5:40 Which word gets picked? 6:47 From guesser to assistant 7:24 The big picture SOURCES 1. Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), chapters 2, 5, 7, 8 and 10. https://web.stanford.edu/~jurafsky/slp3/ 2. Sennrich, R., Haddow, B. & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016. https://doi.org/10.18653/v1/P16-1162 3. Bengio, Y., Ducharme, R., Vincent, P. & Jauvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research 3, 1137–1155. https://www.jmlr.org/papers/v3/bengio... 4. Vaswani, A. et al. (2017). Attention Is All You Need. NIPS 2017. https://proceedings.neurips.cc/paper/... 5. Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020. https://papers.neurips.cc/paper/2020/... 6. Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR 2020. https://iclr.cc/virtual/2020/poster/1428 7. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. https://neurips.cc/virtual/2022/poste... 8. Shannon, C. E. (1951). Prediction and Entropy of Printed English. Bell System Technical Journal, pp. 50–64. https://www.nokia.com/bell-labs/publi... 9. OpenAI. Streaming API responses (API documentation). https://developers.openai.com/api/doc... 10. NLTK Project. Natural Language Processing with Python, chapter 2, section 1.1 "Gutenberg Corpus". https://www.nltk.org/book/ch02.html 11. Pal, K., Sun, J., Yuan, A., Wallace, B. & Bau, D. (2023). Future Lens: Anticipating Subsequent Tokens from a Single Hidden State. CoNLL 2023. https://aclanthology.org/2023.conll-1.37 12. Wu, W., Morris, J. X. & Levine, L. (2024). Do language models plan ahead for future tokens? COLM 2024. https://arxiv.org/abs/2404.00859 13. Linzen, T., Dupoux, E. & Goldberg, Y. (2016). Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. TACL 4, 521–535. https://aclanthology.org/Q16-1037/ ABOUT THIS VIDEO This video is narrated by an AI-generated voice. The script was researched and drafted with AI assistance and checked against the sources above. Visuals are 3D animation, motion graphics and simulations rendered from code written for this channel.