Embedded AI Podcast E02: RAG for Embedded Development: When It Makes Sense (and When It Doesn't)

We explore whether RAG (Retrieval Augmented Generation) still makes sense for embedded development, a topic from Jacob Beningo's Embedded Systems Summit. The concept: ingest technical docs into a vector database, chunk properly, let an LLM retrieve what you need. No manual PDF searching, source citations avoid hallucinations. But it's harder than it looks. Core challenge: chunking. Break documents into pieces small enough to search but large enough to retain context. Getting it wrong means poor results. Mechanics: documents get chunked, each maps to high-dimensional vectors (1024 dimensions), similarity searches in vector space. Fast once set up, but setup is difficult. Is RAG necessary? Modern LLMs have massive context windows. Tools like Claude Code use grep and head for docs on-the-fly. Tree sitter handles code analysis without vector databases. RAG has specific uses: preventing hallucinations and tracing to sources (legal, customer service), structured data (requirements databases, parts catalogs), local LLMs with limited context for privacy/security. Unexpected benefit: sometimes hallucinations help. When an LLM invents API methods that should exist, it's like free design review. Bottom line: RAG is powerful, but for most embedded development, you probably don't need it. Modern LLMs reduced its relevance except specific scenarios. 0:00 Real Humans Making Podcast 1:35 What Is RAG 4:22 Context Windows Matter 7:12 Chunking Is Tricky 13:01 Vector Database Mechanics 17:03 Bespoke RAG Systems 18:52 Alternatives to RAG 22:23 Useful Hallucinations 25:35 RAG Use Cases 31:05 RAG Losing Relevance 36:37 Local LLMs Exception 40:06 Closing Thoughts