LEMAS Seminar by Dr Mustafa Karabag on Strategic Decision making by LLMs

Title: Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions Abstract: Large language models (LLMs) are increasingly used as decision-making agents in multi-agent environments. To act effectively under uncertainty, they must infer hidden information, decide what information to reveal or conceal, and translate their beliefs into actions. Yet our understanding of these strategic capabilities remains limited. In this talk, I examine strategic decision-making through the lens of information control. I first study language-based multi-agent interactions, showing that LLM agents often reveal excessive information in situations where successful behavior requires careful concealment. Through theoretical and empirical analyses of a hidden-identity game, I characterize the strategic tradeoffs between information concealment and revelation, show that LLMs perform far below simple benchmarks, and demonstrate how steering their internal representations can induce more strategically effective behavior. I then analyze LLM internal representations in incomplete-information games and uncover two fundamental failures in strategic reasoning: an observation-belief gap, where internal beliefs are more accurate than verbalized beliefs but deteriorate over extended interactions, and a belief-action gap, where models fail to act effectively even when their internal beliefs are informative. Together, these results reveal systematic weaknesses in how LLMs represent, communicate, and utilize information in strategic settings. I conclude by discussing open challenges and future directions for strategic reasoning and information control in LLMs. Bio: Dr. Mustafa Karabag is a Research Associate in the Oden Institute for Computational Engineering & Sciences at the University of Texas at Austin and an incoming Assistant Professor in the Department of Electrical and Computer Engineering at the University of South Florida. He earned his M.S. and Ph.D. degrees in Electrical and Computer Engineering from the University of Texas at Austin in 2019 and 2023, respectively. In 2025, he also served as a Lecturer in the Department of Aerospace Engineering and Engineering Mechanics at the University of Texas at Austin. His research develops theoretical foundations and algorithms for control and learning in strategic autonomous systems, integrating control theory, artificial intelligence, optimization, and game theory.