93% of My RAG Training Data Came From the Wrong Retriever

A small RAG model was fine-tuned for seven rounds, and every dataset check passed every time. But about 93% of its training rows had been built with an LSA retriever, while the product served answers with nomic-embed-text vectors. The two retrievers barely agreed: for "forgiving your enemies", one returned verses about forgiveness and the other returned verses about hating enemies. Here is how that hid, why no per-row check could ever see it, and the config pattern that stops it. What's covered: • Two indexes in one SQLite file: LSA (no model, no server) vs nomic-embed-text (needs Ollama) • How a default flag and a hardcoded provider built the training set with the wrong retriever • The measurement: Jaccard vs the serving retriever of 0.038 on the main rows and 0.077 on the answer rows • Why per-row validation passes: the defect is in the joint distribution • Why a full fine-tune got the best loss and worse behaviour than LoRA (44.4% vs 52.3% refusal) • The silent fallback: Ollama bound to a tailnet address, so "auto" quietly used LSA • Probing for a daemon properly, and why reachable is not available • Three more mismatches: a chat template, unseen inline verse numbers, and k=6 vs k=8 • The asymmetric-prefix trap in nomic and E5 embeddings • A RetrieverSpec pattern: one spec, strict builds, a fingerprint on every row, a Jaccard audit • Repairing a wrong dataset: regenerate, re-judge, or abstain, never just swap passages ⏱️ Chapters 00:00 Train-Serve Mismatch 01:04 Two Retrievers, One Corpus 03:00 Why Every Check Passed 04:35 The Fallback Was Silent 06:09 Three More Mismatches 07:39 The Config Pattern 09:07 Fixing a Wrong Dataset 📖 The full guide (the measurements, the probe order, the RetrieverSpec code, the audit, the repair options): → RAG Train-Serve Mismatch: The Retriever That Built the Data Must Serve: https://www.bigiron.cc/guides/rag-tra... 🛠️ Gear (affiliate — same price to you, supports the channel): • 12 GB-class GPU for local embeddings and LoRA fine-tunes → https://www.amazon.com/s?k=12gb+graph... • NVMe SSD for datasets and vector indexes → https://www.amazon.com/s?k=2tb+nvme+s... • 64 GB RAM for a local-model box → https://www.amazon.com/s?k=64gb+ddr5+... As an Amazon Associate we earn from qualifying purchases. Affiliate links — same price to you, supports the channel. 👉 Subscribe for self-hosted AI + homelab, with new videos every day. #RAG #FineTuning #LoRA #Embeddings #Ollama #LocalAI #LLM #MachineLearning #SelfHosted #Homelab