Beyond the Turing Test: Evaluating Conversational Agents | Dr. Roderick Boone

What makes a conversational agent successful? In this video, Dr. Roderick Boone presents a framework developed from lectures, readings, and coursework to help users and developers understand, design, and evaluate conversational agents. Tracing their evolution from ELIZA to modern large language models, he distinguishes two purposes: completing a defined task and sustaining a worthwhile social conversation. Each requires different strategies and measures of success. The discussion covers: • Dialogue properties, including turn-taking, grounding, and clarification. • Shared challenges involving context, privacy, transparency, bias, and hallucinations. • Evaluation concerns such as data contamination and sensitivity to wording. • The limitations of human resemblance as a measure of conversational quality. • Task success, engagement, coherence, conversational depth, and topical diversity. Drawing on Venkatesh et al.’s evaluation research, Dr. Boone explains why systematic assessment should combine multiple measures with human judgment. The central message: evaluation should reflect the agent’s purpose and the quality of the experience it delivers. Explore the framework shared with this video and join the discussion: What would you measure to determine whether a conversational agent is effective? #ConversationalAI #ResponsibleAI #NLP #AIEvaluation