Ep 31 | Cut LLM Costs 90% Without Killing UX: Caching, Routing, Batching

Episode 31 of the series "AI Engineering for Experienced Developers". Running LLM apps in production often means watching latency and API bills spiral at the same time. In this episode we break down five concrete techniques experienced engineers use to fix both: prompt caching, model routing, request batching, streaming UX tricks, and knowing when a smaller model is actually the right call. This is Episode 31 in our 'AI Engineering for Experienced Developers' series, part of Module 6 on Production and LLMOps. If you're moving from prototype to production with LLM apps, this episode gives you a practical mental model for trading off cost, latency, and quality. Chapters 0:00 Latency and Cost Optimisation for LLMs 0:48 Why Latency and Cost Are Linked 1:39 Prompt Caching 2:29 Model Routing 3:18 Batching Requests 4:06 Streaming for Perceived Speed 4:54 Smaller Models by Default 5:40 Combining the Techniques 6:28 A Practical Rollout Order 7:12 Common Pitfalls 7:56 Recap and Next Steps #LLMOps #AIEngineering #PromptCaching #ModelRouting #LLMCost