Beyond the Demo: A Structured Approach to AI Vendor Assessment/ Part I

AI Vendor Risk Management: How to Assess AI Vendors That Are a Black Box Your company just signed a contract with an AI vendor. The demo looked great, the accuracy numbers were convincing, and the hallucination rate seemed low. But what do you actually know about what's happening inside that model? In this session, Valeri Milke (CEO, VamiSec GmbH) and Varinder Kumar (Information Security Consultant and AI Security & Governance Officer at VamiSec) discuss why traditional third-party risk management falls short for AI solutions. Varinder presents a practical framework for assessing AI vendors. What you'll learn: The black box problem: the eight things you can't see when evaluating an AI vendor, from model architecture and training data to drift, retraining pipelines and hidden third-party components. Evidence instead of promises: why an AI Bill of Materials (AI BOM) matters, what to demand instead of vendor benchmarks, and which AI-specific requirements belong in your contract. 12 assessment domains: how the framework scores vendors across performance and accuracy, explainability, bias and fairness, data governance and privacy, security and adversarial robustness, human oversight, supply chain risk, regulatory compliance and more. 22 practical tests: why you should test vendors against your own data, how to run prompt injection tests (including hidden instructions in PDFs), how to verify data deletion, and why you should check whether you can actually stop a model mid-session. Standards and regulations: ISO/IEC 42001, the EU AI Act and the NIST AI RMF, each explained in plain terms, plus why an integrated management system beats working in silos. When vendors fail: why "enterprise grade" is a marketing term with no fixed meaning, why a failed test isn't automatically a deal breaker, and how remediation clauses with real deadlines strengthen your negotiating position. Key takeaway: AI vendor risk management doesn't replace your existing third-party risk process. It adds a layer on top of it, at the company level and at the level of the AI system itself. Follow-up session coming soon: we'll run real-world tests against well-known AI solutions using this framework. How does your organization assess AI vendors today? Let us know in the comments. Subscribe for more on AI governance, information security and compliance. VamiSec GmbH: https://vamisec.com https://vamigrc.com/ #AIGovernance #VendorRiskManagement #ThirdPartyRisk #AIRisk #ISO42001 #EUAIAct #NISTAIRMF #PromptInjection #AISecurity #GRC #Cybersecurity #Compliance