EP 02B — Build a Trigram Language Model in Python

Build a trigram language model in Python, then give it even more context and measure what changes. In this coding episode of Building an LLM From Scratch, we turn the theory into working code: context windows, sparse count dictionaries, on-demand smoothing, negative log-likelihood, and character-by-character name generation. We start with anna and ava, then compare models with one to five context characters on the same names dataset used in Episode 01. Does extra context improve predictions, or leave each row with too little evidence? We compare training and validation loss, inspect rare and unseen contexts, choose settings using validation, and report the fixed models on test. Run the complete notebook (code, charts, and saved outputs): https://github.com/HussainAbuwala/llm... Code companion guide - setup and explanations: https://github.com/HussainAbuwala/llm... Download or clone the whole repository so the notebook can find its dataset and chart helpers: https://github.com/HussainAbuwala/llm... Episode 02A theory companion (PDF): https://github.com/HussainAbuwala/llm... Theory canvas - all 29 frames (PDF): https://github.com/HussainAbuwala/llm... Chapters: 00:00 From the theory to Python: setup and context windows 05:00 Sparse counts and smoothed probabilities 12:47 From toy samples to the larger dataset 14:47 Comparing context lengths and choosing smoothing 18:47 Reading training and validation loss 22:47 Sparse evidence: rare and unseen contexts 26:47 Final test comparison and takeaways What you'll build: • A count-based model with arbitrary context length • Probabilities computed from sparse counts without creating unseen training rows • Name evaluation and weighted sampling with explicit END handling • A validation comparison across context lengths and smoothing values • Charts showing prediction loss and the evidence behind each context A trigram uses two previous characters to predict one next token. Three context characters make a 4-gram. Our experiment goes beyond trigrams to test the tradeoff discussed in theory. Among the tested settings, validation selected three context characters with k = 0.1. Its test NLL is 2.129940 nats per prediction, compared with 2.460499 for the bigram baseline. Both use the same 21,053 test predictions, including END. These results describe this dataset, split, and add-k model family; they do not establish a universal best context length. The test split is the same one previously reported in Episode 01. It is excluded from this episode's training and model selection, but it is not a newly collected independent dataset. Backoff and interpolation are theory topics; this coding comparison evaluates add-k models. The model uses Python's standard library. Jupyter presents the notebook and Matplotlib draws the charts. Basic Python knowledge and the Episode 01 bigram concepts are helpful. Dataset credit: names.txt from Andrej Karpathy's makemore repository. The dataset license and pinned source information are preserved in our repository. We use 29,494 unique name spellings, each equally weighted. https://github.com/karpathy/makemore #Python #LLMFromScratch #MachineLearning