How ChatGPT Actually Works: The Math, Step by Step

ChatGPT writes every answer one word at a time by guessing the next word. Here's the math behind each step, from tokens and embeddings to attention, 96 layers and training, and why guessing well ends up looking like thinking. Chapters 0:00 Intro 0:23 Text becomes numbers 0:58 Numbers become meaning 1:38 Guessing the next word 2:32 Attention 3:22 Stacking layers 4:09 Training 4:50 Why guessing works 5:30 What this means for you The 50-second version:    • AI Doesn't Write Sentences. Here's What It...   Sources: Token IDs: OpenAI's tiktoken library, cl100k_base (GPT-4's tokenizer) Word vectors (king − man + woman ≈ queen): Mikolov et al., 2013 Attention Is All You Need: Vaswani et al., 2017 GPT-3 sizes, training tokens and compute: Brown et al., 2020, "Language Models are Few-Shot Learners" Fine-tuning from human ratings: Ouyang et al., 2022 (InstructGPT) Probabilities, attention weights and vector values shown on screen are example numbers. #ChatGPT #AI #MachineLearning #ComputerScience #Math