Advanced RAG, End to End: Build an AI Research Assistant

Build an AI research assistant that finds papers, answers from their abstracts, and checks every citation before it ships. In this advanced RAG case study we go step by step over arXiv abstracts — and justify every decision: licensing, data cleaning, filtered vector search, query planning, a safe count route, hybrid retrieval with RRF and re-ranking, HyDE as an experiment, citation checks, an evidence gate, and an evaluation you can trust (own test set, pooling, nDCG, McNemar). No invented results: the ablation table is a template for your own run. Chapters 0:00 Intro: keyword vs neural on scientific search 1:24 Step 1 — Scope: four question types, the worst failure 3:27 Step 2 — Data selection and licensing (CC0 metadata, no full text) 6:26 Step 3 — Cleaning: dates, versions, withdrawn papers 8:09 Step 4 — What to embed (and why no chunking here) 10:22 Step 5 — The index: pgvector, weighted full text, overfiltering 12:23 Step 6 — Planning the query: routes, validation, safe count SQL 14:49 Step 7 — Hybrid retrieval, RRF, re-ranking, HyDE 17:28 Step 8 — Evidence-first answers and the evidence gate 19:46 Step 9 — Evaluation: own test set, pooling, nDCG, McNemar 23:35 Step 10 — Ship and operate: incremental harvest, versions 25:34 Recap: the decision checklist Sources (every number on screen) BEIR benchmark — Thakur et al., 2021 (arXiv:2104.08663): SciFact nDCG@10 BM25 0.665, DPR 0.318, BM25+CE 0.688; BM25+CE beats BM25 on 16/18 datasets arXiv Terms of Use for APIs (metadata under CC0; e-print copyright; rate limits) — info.arxiv.org/help/api/tou.html arXiv OAI-PMH help (commercial reuse of metadata; incremental harvesting) — info.arxiv.org/help/oa arXiv: withdrawals — info.arxiv.org/help/withdraw.html Contextual Retrieval — Anthropic, Sep 2024 Reciprocal Rank Fusion — Cormack, Clarke & Büttcher, SIGIR 2009 HyDE — Gao et al., 2022 (arXiv:2212.10496) Corrective RAG — Yan et al., 2024 (arXiv:2401.15884) pgvector 0.8.0 iterative index scans; BAAI bge-base-en-v1.5 / bge-reranker-base model cards (MIT) PostgreSQL text search docs; statsmodels mcnemar docs Worked McNemar p-values computed with an exact two-sided binomial test Data: paper metadata from arXiv (via the Kaggle arXiv dataset), used under CC0 1.0. Full texts are not used. This video is not affiliated with or endorsed by arXiv or Kaggle. Note: the ablation table is an empty template — no results for this system are claimed. Series: RAG Query Processing · RAG Evaluation · Advanced RAG Patterns · RAG Case Study (beginner) · this video Build it and post your ablation table in the comments. #RAG #LLM #AIEngineering Tags advanced RAG, RAG case study, hybrid search, BM25 vs dense retrieval, reciprocal rank fusion, RRF, re-ranking, cross encoder, pgvector, pgvector filtering, iterative index scan, HyDE, corrective RAG, CRAG, text to SQL safety, citation hallucination, RAG evaluation, nDCG, McNemar test, arXiv dataset, research assistant RAG, production RAG, LLM engineering, AI engineering