How to Test an AI Agent

One passing test tells you surprisingly little: after a single pass, the honest 95% range for an agent's real pass rate is 21% to 100%. In this video we build an evaluation harness for Ops, the agent from this series, in plain Python. It covers: tasks and trials, balanced between "should act" and "should not act"; outcome, path and output graders; transcripts; Wilson confidence intervals; comparing two versions per task; model-as-judge, with two bias tests; a CI gate. Important: every run in this video uses a simulated model whose mistake rates we chose on purpose. It teaches the method; it makes no claim about how any real model behaves. Chapters 0:00 One pass proves almost nothing 1:06 The words: task, trial, grader, transcript, outcome, harness 1:57 The task set (should act / should not act) 3:30 Graders: outcome, path, output 5:28 The harness 6:44 The simulated model 7:36 The first run 8:27 pass@k vs pass^k 10:40 Read the transcripts (the grader was wrong) 11:42 How sure are we? Wilson intervals 13:14 Comparing two versions: the average hid a regression 15:08 Model as judge: padding and swap tests 17:37 Evals in CI, recap, next Notes: All code is standard-library Python, and the harness has its own tests, which pass. The model is simulated, and the mistake rates are chosen by us. The judge is a deliberately biased stand-in, not a model. The hand labels were written for the demo. AI Agents series: 1) First Principles · 2) Build an AI Agent · 3) MCP Server from Scratch · 4) Agent Memory · 5) this video · next: the end-to-end build. How many trials do you run before you trust an agent change? #AIAgents #LLM #Python Tags AI agent evaluation, agent evals, LLM evaluation, evaluate AI agents, pass@k, pass^k, tau-bench, LLM as a judge, LLM judge bias, Wilson interval, confidence interval, regression testing AI, AI agents, build AI agent, Python AI agent, MCP, AI engineering, LLM engineering, Anthropic evals