tt syd 26 трек1 8

Artificial intelligence is rapidly moving into real world systems, yet the way we test software has barely changed. Traditional testing assumes that a given input should produce a predictable and correct output. AI systems challenge this assumption. Large language models and AI driven workflows produce probabilistic responses that may vary between runs and depend heavily on prompts, context, and retrieved information. This creates a new challenge for testers. When multiple responses may be acceptable, how do we define quality and verify system behaviour? This talk explores why traditional testing approaches struggle with AI systems and how quality engineering must evolve. Using examples from AI assisted development tools and retrieval based systems, we examine common failure patterns across model behaviour, context retrieval, and orchestration layers. The session introduces practical evaluation techniques that testers can apply to AI systems. Instead of relying only on deterministic assertions, teams can use behavioural metrics and reference based evaluation methods such as semantic similarity, grounding checks, BLEU, and ROUGE to measure response quality, consistency, and factual alignment. Attendees will leave with a clear mental model for testing AI systems and practical evaluation techniques they can begin applying in real AI powered products.