How AI reads a sentence: attention, from scratch

"I withdrew money from the bank." "We sat by the river bank." You know which bank is which. A language model gets the same token (3331) and the same vector for both, so whatever tells them apart has to come from the other words. That is attention. We follow GPT-2, whose every detail is public: text to tokens, tokens to vectors, then attention computed by hand with small numbers you can check: queries, keys, the dot product, dividing by the square root of the length, softmax, values, and the causal mask. Then heads, blocks, the network after attention, the next-token loop, and how training sets the weights. Chapters 0:00 Two banks 0:39 Tokens 1:13 Vectors 2:05 The idea of attention 2:40 Query, key and the dot product 3:46 Softmax and values 4:31 Every token at once: the formula 5:29 Heads and blocks 6:22 The next token, and training 7:04 Recap Token IDs from OpenAI's tiktoken (gpt2 encoding). The worked example uses toy 2-number vectors; GPT-2 uses 64 per head. Formula: Vaswani et al., "Attention Is All You Need" (2017). #attention #transformer #LLM #machinelearning #GPT #deeplearning