Is Your AI Agent Ready? Evaluations, Red Teaming and Guardrails in Microsoft Foundry

Is your AI agent ready for real customers? In part 1 we built Penny for Blue Penguin Bakery, in part 2 she learned the bakery's documents with Azure AI Search, in part 3 she got a teammate, the Orders agent, over A2A, and in part 4 a Kitchen agent started working by itself, on events and on a schedule. All of it works. But working is not the same as ready. In this last video of the series, we test, attack, protect and watch the bakery's agents, in the Foundry portal and the Azure portal. Then we run the whole bakery from a customer's order to the kitchen's morning check, and clean up. Every click is shown, and every idea is explained along the way. What you'll learn • Evaluations in Microsoft Foundry: synthetic test questions, the suggested evaluators, a judge model, and reading the reasons behind the scores • Analyze Results: grouping what went wrong into clusters • Red teaming an agent: risk categories, attack strategies (Base64, Flip, Jailbreak) and the attack success rate • Guardrails: blocking harmful content, indirect prompt injections and sensitive data, like a credit card number, on a single agent • Application Insights for Azure Functions, and one order as a single end-to-end trace • Foundry's Operate and Monitor pages, and Evaluation Alerts • An Azure Monitor alert on a Service Bus dead-letter queue • What it costs, and a budget with an email alert • One full run: an order through Penny, the Orders agent, the functions, the queue and the Kitchen agent • Cleaning up, including what a resource group delete leaves behind 💻 Code and settings: https://github.com/infra2ai/agents-in... The evaluation, red team, guardrail, alert and budget settings used in the video, and the whole bakery's functions and agent instructions as part 4 left them. 📺 Part 1, building Penny:    • Build Your First AI Agent on Azure – Step-...   📺 Part 2, Azure AI Search for AI agents:    • Azure AI Search for AI Agents – Give Your ...   📺 Part 3, multiple AI agents as a team (A2A):    • Multiple AI Agents That Work as a Team: A2...   📺 Part 4, event-driven and scheduled AI agents:    • Event-Driven AI Agents on Azure: Service B...   📺 The whole series:    • Build AI Agents on Azure, End to End | Blu...   💻 Part 1 code: https://github.com/infra2ai/first-ai-... 💻 Part 2 code: https://github.com/infra2ai/rag-on-azure 💻 Part 3 code: https://github.com/infra2ai/multi-age... 💻 Part 4 code: https://github.com/infra2ai/kitchen-o... 📍 Region: in this video the bakery is set up again in Sweden Central, because cloud red teaming only runs in East US 2, France Central, Sweden Central, Switzerland West and US North Central. If you followed parts 1 to 4 in Switzerland North, as I did, red teaming isn't available there. Azure AI Search can stay in another region. 💰 The one thing that charges while nothing is happening is Azure AI Search, about 10 cents an hour for as long as it exists. Evaluations and red teaming use model tokens, which is small for a demo this size. Application Insights, Azure Functions, Service Bus on the Basic tier and Table storage are a few cents. A budget and its email alert are free; a metric alert rule is about 10 cents a month. Following along costs a few dollars at most, as long as you delete the resource group at the end. We do it together in the video. 🧹 After you delete the resource group, two things stay behind: • The Foundry resource is kept, soft-deleted, for 48 hours, and its name stays taken. To reuse the name, list it and purge it: az cognitiveservices account list-deleted, then az cognitiveservices account purge -l location -g resource group -n foundry resource name • The agents' identities live in Microsoft Entra ID, not in the resource group. Under App registrations and Enterprise applications, look for names that start with your Foundry resource's name (they end in AgentIdentity or AgentIdentityBlueprint) and delete them. ⏱️ Chapters 0:00 Welcome 0:54 The whole series so far 1:36 The problem: working is not ready 2:11 What we'll do today 3:27 Evaluations: is it right? 4:56 Evaluating the three agents 8:12 Red teaming: can it be tricked? 9:46 Red teaming Penny 12:25 Guardrails: what stops it? 13:48 A guardrail for Penny 16:37 Monitoring: will I know? 17:20 Application Insights for the functions 18:17 Watching the agents in Foundry 19:56 An alert on the queue 21:49 What it costs 22:41 A budget with an alert 23:53 The full run: the plan 24:33 An order, from customer to kitchen 26:30 The kitchen's morning check 27:36 What the monitoring saw 29:06 Clean up 30:45 Recap, habits and thank you 🙏 This is the last video of the series: five parts, one bakery. Thank you for building it with me. If it helped you, please like the video, leave a comment, and subscribe. And tell me in the comments which series you'd like to see next: a topic, and the Azure services you care about. A new series is coming, so watch out for it! 🎙️ Narration: AI voice (Azure AI Speech). #Azure #AIAgents #MicrosoftFoundry