Using Large Language Models for Evaluation: Opportunities and Limitations

Talk Title: Using Large Language Models for Evaluation: Opportunities and Limitations Speaker: Prof. Emine Yilmaz Date: May 27, 2026 Abstract: Large Language Models (LLMs) have shown significant promise as tools for automated evaluation across diverse domains. While the use of LLMs for evaluation offers substantial advantages—potentially reducing reliance on costly and subjective human assessments—the adoption of LLM-based evaluation is not without challenges. In this talk, we discuss both the transformative potential and the inherent limitations of using LLMs for evaluation tasks. In particular, we highlight challenges such as bias and variability in judgments. We also explore how LLMs can augment traditional evaluation practices while emphasizing the need for a cautious and informed approach to their use.