LEMAS Seminar by Professor Kaiqing Zhang on Learning-to-communicate in MARL

Title: Towards Understanding Learning-to-Communicate in Multi-agent RL: An Information-Structure Perspective Abstract: Learning-to-Communicate (LTC) in partially observable environments has emerged and received increasing attention in deep multi-agent reinforcement learning, where the control and communication strategies are jointly learned. On the other hand, the impact of communication has been extensively studied in control theory. In this paper, we seek to formalize and better understand LTC by bridging these two lines of work, through the lens of information structures (ISs). To this end, we formalize LTC in decentralized partially observable Markov decision processes (Dec-POMDPs) under the common-information-based framework from decentralized stochastic control, and classify LTC problems based on the ISs before (additional) information sharing. We first show that non-classical LTCs are computationally intractable in general, and thus focus on quasi-classical (QC) LTCs. We then propose a series of conditions for QC LTCs, violating which can cause computational hardness in general. Further, we develop provable planning and learning algorithms for QC LTCs, and show that some examples of QC LTCs satisfying the above conditions can be solved with quasi-polynomial time and samples. Along the way, we also establish some relationship between (strictly) QC IS and the condition of having strategy-independent common-information-based beliefs (SI-CIBs), as well as solving Dec-POMDPs without computationally intractable oracles but beyond those with the SI-CIB condition, which may be of independent interest. Time permitting, we will also discuss the extension to linear-quadratic control settings. Bio: Kaiqing Zhang is currently an Assistant Professor at the Department of Electrical and Computer Engineering and the Institute for Systems Research, with joint appointment at the Department of Computer Science, at the University of Maryland, College Park. His research interests lie broadly in Systems and Control Theory, Game Theory, Reinforcement/Machine Learning, Robotics, Computation, and their intersections. His work has been recognized by several awards, including Simons-Berkeley Research Fellowship, Coordinated Science Lab Thesis Award, AAAI New Faculty Highlights, NSF CAREER Award, AFOSR YIP Award, Donald P. Eckman Award, Faculty Research Awards from Amazon, Cisco, Open Philanthropy, and JP Morgan, and Best Paper Awards from ICML, ISDG, and IEEE RAS. He also received the George Corcoran Memorial Award for Teaching and Educational Leadership.