Agentic Workflows for Computer Systems Research (paper review)

Discussed papers: • Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia et al. “Barbarians at the gate: How ai is upending systems research." https://arxiv.org/abs/2510.06189 • Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Shubham Agarwal, Mert Cemri, Bowen Wang et al. “Let the Barbarians In: How AI Can Accelerate Systems Performance Research.” https://arxiv.org/abs/2512.14806 • Rohit Dwivedula, Divyanshu Saxena, Sujay Yadalam, Daehyeok Kim, and Aditya Akella. “Vulcan: Instance-Optimal Systems Heuristics Through LLM-Driven Search.” https://arxiv.org/abs/2512.25065 ADRS abstract: Artificial Intelligence (AI) is beginning to transform the research process by automating the discovery of new solutions. This shift depends on the availability of reliable verifiers, which AI-driven approaches require to validate candidate solutions. Research focused on improving systems performance is especially well-suited to this paradigm because system performance problems naturally admit such verifiers: candidates can be implemented in real systems or simulators and evaluated against predefined workloads. We term this iterative cycle of generation, evaluation, and refinement AI-Driven Research for Systems (ADRS). Using several open-source ADRS instances (i.e., OpenEvolve, GEPA, and ShinkaEvolve), we demonstrate across ten case studies (e.g., multi-region cloud scheduling, mixture-of-experts load balancing, LLM-based SQL, transaction scheduling) that ADRS-generated solutions can match or even outperform human state-of-the-art designs. Based on these findings, we outline best practices (e.g., level of prompt specification, amount of feedback, robust evaluation) for effectively using ADRS, and we discuss future research directions and their implications. Although we do not yet have a universal recipe for applying ADRS across all of systems research, we hope our preliminary findings, together with the challenges we identify, offer meaningful guidance for future work as researcher effort shifts increasingly toward problem formulation and strategic oversight. Vulcan abstract: Resource-management tasks in modern operating and distributed systems continue to rely primarily on hand-designed heuristics for tasks such as scheduling, caching, or active queue management. Designing performant heuristics is an expensive, time-consuming process that we are forced to continuously go through due to the constant flux of hardware, workloads and environments. We propose a new alternative: synthesizing instance-optimal heuristics -- specialized for the exact workloads and hardware where they will be deployed -- using code-generating large language models (LLMs). To make this synthesis tractable, Vulcan separates policy and mechanism through LLM-friendly, task-agnostic interfaces. With these interfaces, users specify the inputs and objectives of their desired policy, while Vulcan searches for performant policies via evolutionary search over LLM-generated code. This interface is expressive enough to capture a wide range of system policies, yet sufficiently constrained to allow even small, inexpensive LLMs to generate correct and executable code. We use Vulcan to synthesize performant heuristics for cache eviction and memory tiering, and find that these heuristics outperform all human-designed state-of-the-art algorithms by upto 69% and 7.9% in performance for each of these tasks respectively. 00:00:00 Papers discussed 00:01:18 Agentic discovery 00:02:42 Agentic workflows for systems research 00:01:18 AI-driven research for systems (ADRS) 00:05:49 ADRS assumptions 00:07:15 Results overview 00:08:18 Let the barbarians in 00:09:37 LLM scaffold 00:11:05 Takeaways from scaffold evas 00:13:15 Breaking domain boundaries 00:15:01 Failure modes 00:16:17 Best practices and tips 00:19:42 Shortcomings of ADRS 00:20:30 Intro to Vulcan 00:22:22 Narrower agent-researcher interface 00:28:11 Defining an instance 00:24:18 Vulcan evaluation 00:35:32 Summary 00:37:58 Implications on research and reflection #agenticai #research #computerscience #highereducation