LLM Evals Interview 2026: The Score Went Up. A Case Broke.

Twelve interview questions on evaluating and securing LLM apps: eval types, calibrating an LLM judge, golden datasets, regression-testing a prompt change (an average that went up while a case broke), PII redaction, prompt injection that a heuristic misses, least-privilege tool policies, logging, cost limits, incident response and red-teaming. Everything runs in a small plain-Python lab. Part 9 of AI Engineer Interview Kit 2026. Chapters 0:00 Intro 0:41 1. Design an eval suite: which grader for what? 1:28 2. Would you trust an LLM judge? Calibrate it 2:27 3. What belongs in a golden dataset, and how does it grow? 3:10 4. The pass rate went up. Is the change safe? 4:10 5. Where do guardrails sit, and what do cheap rules miss? 5:13 6. Defend an agent against indirect prompt injection 6:20 7. Stop sensitive data leaking via prompts and logs 7:05 8. Limit what an agent can do 7:50 9. What do you log to replay a failure? 8:30 10. Cap cost and abuse 9:09 11. Incident: the agent took a wrong action. Walk me through it 9:50 12. Red-team it: what happens to the findings? Code for this video: https://github.com/codeandagents/inte... Versions used: Python 3.14 (standard library only) Sources: https://genai.owasp.org/resource/owas... https://arxiv.org/abs/2306.05685 https://www.anthropic.com/engineering... About the channel: tutorials by a software architect with 20+ years of designing and building highly scalable systems. Narration uses an AI voice; the code, the runs on screen and the explanations are the author's own.