The Missing Clause: Search Engines and Information Retrieval

From Callimachus, concordances and Vannevar Bush to Luhn, Cleverdon's Cranfield experiments, Salton's vector space model, Spärck Jones's IDF and TREC; postings, Porter stemming, compression, WAND, Lucene and Elasticsearch; the probability ranking principle and BM25 from Okapi at TREC-3; pooling, precision, recall, MRR, MAP, nDCG, interleaving; PageRank, HITS and spam; query understanding and Rocchio; RankNet, LambdaMART; word2vec, BERT, cross-encoders, DPR, ColBERT, SPLADE, MTEB; LSH, product quantisation, FAISS, HNSW, DiskANN; reciprocal rank fusion; structure-aware chunking and version graphs; multilingual retrieval; knowledge graphs; RAG, lost in the middle, RAGAS and calibrated LLM judges; permissions and prompt injection. Cross-references: ep193 (LSM trees, tail latency). Chapters 00:00:00 Cold open: the missing clause 00:02:32 From catalogues to Cranfield 00:06:41 The inverted index 00:10:55 BM25: the probabilistic ranking 00:14:17 Measuring relevance 00:18:37 The web, PageRank and spam 00:21:52 Query understanding 00:23:51 Learning to rank 00:26:19 Neural retrieval: from words to vectors 00:31:05 Nearest neighbours at scale 00:34:21 Hybrid retrieval and reranking 00:36:57 Chunking, structure and versions 00:39:56 Beyond English text 00:42:49 Structured questions and knowledge graphs 00:44:58 Retrieval-augmented generation 00:48:09 Evaluating RAG honestly 00:51:09 Permissions and prompt injection 00:52:34 Keeping the index alive 00:54:44 Learning from the lawyers 00:55:57 Search as an industry 00:57:52 Desktop search and the personal archive 00:59:11 The latency and cost budget 01:01:30 Dawn: the scorecard Created by Sankalp Thakur. A Transient Labs production. Website: https://transientlabs.ai GitHub: https://github.com/sankalpsthakur X / Twitter: https://x.com/sankalpsthakur LinkedIn:   / sankalpthakur   Hugging Face: https://huggingface.co/sankalpsthakur Scripts generated with Claude Opus. Synthetic narration generated with Kokoro. Reviewed, listened to, and approved by Sankalp Thakur.