The AI Demo Looked Great. Then Real Users Showed Up.

Someone on your team is already using an AI tool nobody approved. And the next vendor demo you sit through will look great whether or not the product holds up on your data. Stella Liu, Head of AI Applied Science and Lead Data Scientist in ASU's AI Acceleration group, explains what AI evaluation actually measures and why products that pass a quick vibe test fall apart once real users arrive. We cover the vendor red flags she watches for, the first question to ask any AI vendor, how to build guardrails into procurement without slowing your team down, and the one question to ask your own team tomorrow morning. Chapters 0:00 Vibe tested, shipped, then broken 0:36 Introduction 1:38 Somewhere between AI hype and fear 3:14 Finding Shadow AI without a witch hunt 4:28 What AI evaluation means in practice 5:11 Performance, quality and safety 6:35 Why AI pilots stall: deterministic vs probabilistic software 8:54 The Photoshop test and the shiny demo problem 10:40 Red flags in AI vendors 14:06 The first question to ask any AI vendor 15:24 Guardrails in procurement without slowing the team 18:15 When compliance lags behind AI 19:32 The first question to ask your team 20:52 Feeling more productive vs being more productive 22:30 Where to find Stella Connect with Stella LinkedIn:   / wenxingl   Data Science x AI newsletter: https://datasciencexai.substack.com AI Evals and Analytics course on Maven: https://maven.com/ai-evals-and-analyt... Show notes and full transcript: https://theaihat.com/how-to-evaluate-... Mentioned in this episode Shadow AI - what it is and how to lean into it: https://theaihat.com/shadow-ai If you'd like to see where your team's hours are actually going before you change anything, take the AI Opportunity Score. It takes about three minutes, shows you how many hours a week your team could get back, and gets you a starter crew member built for the biggest one. https://theaihat.com/ai-opportunity-s... Listen on your favorite podcast app: https://theaihat.com/podcast