Multi-agent AI measures of methodological complexity help forecast reproducibility | Daniele Fanelli

Daniele Fanelli argues that a study's reproducibility should decrease as the complexity of its methods increases, and presents a way to measure that complexity with LLMs. Earlier hand-coded graphs of protocols from the Brazilian Reproducibility Initiative already tracked replicators' forecasts. A four-agent pipeline (Claude Sonnet 4.6) now rewrites a protocol as steps, turns them into a knowledge graph with attributes, validates it and standardises terms across the corpus. Each graph gives a description length and the surprisal of its nodes and attributes. In the Brazilian data, complexity correlates with expert-rated difficulty and negatively with a continuous reproducibility measure, mainly between methods. In 1,861 FORRT replication studies, descriptive complexity predicts lower replication success, but node surprisal predicts higher success. Fanelli frames the work as a meta-scientific question about what causes irreproducibility. Speaker: Daniele Fanelli (Heriot-Watt University and LSE) Authors: Daniele Fanelli and Yingfang Yuan Talk title: Multi-agent AI measures of methodological complexity help forecast study reproducibility Session: Session 6 · Predicting replicability Recorded at AI in Metascience 2026, an online conference organised by FORRT (Framework for Open and Reproducible Research Training), 28–30 September 2026. Day 2, Tuesday 29 September: Replications and reproductions. Programme and abstracts: https://forrt.org/ai-metascience/ FORRT: https://forrt.org