From Supervising AI to Building Systems for It

Eleanor Berger is a member of the technical staff at Jimini Health, where she helps build evidence-based AI for behavioral health. Through Agentic Ventures, she also teaches engineers and teams how to work with agentic AI. Dan and Eleanor discuss the shift from supervising agents locally to delegating work to asynchronous cloud environments. Eleanor argues that stepping away from a running session forces clearer specifications and makes it easier to manage parallel tasks. Sandboxed execution, controlled access to external systems, and repository instructions give agents a workable environment. For Eleanor, the strongest controls are deterministic. She describes moving to test-driven development and giving agents static analysis, tests, and checker scripts that expose mistakes without another model having to judge them. She also explains why she increasingly favors statically typed languages for agent work. The engineer's responsibility shifts toward designing the system and deciding how to verify its behavior. Chapters: 00:00 Introduction 00:58 The flip: from supervising AI to getting out of its way 03:14 Cloud-based agents vs. rolling your own 06:08 The primitives every agent system needs 07:43 Why async delegation beats local babysitting 11:02 Writing specs: Codex, Repo Prompt, and markdown 12:21 Guardrails: AGENTS.md, skills, and deterministic checks 14:19 Going fully test-driven 17:19 How engineers really adopt (and hide) AI 19:38 Getting better through deliberate practice 21:12 From experiment to reusable skill to library 24:26 Choosing a language: Python, TypeScript, Go 26:56 Supply chain risk and distributing specs 29:06 Beyond 'junior': curiosity over tenure 31:03 Systems thinking as the durable skill 38:02 Where Eleanor still doesn't trust AI 39:40 Not missing the keyboard 42:43 Keeping up with a fast-moving field 44:50 What teaching reveals 48:27 Verification as the real bottleneck 50:41 Team size and open source at scale 55:48 Closing: take agents seriously