PaperRobust: a multi-agent framework for assessing analytical robustness | Qian Ma

Qian Ma introduces PaperRobust, an ongoing project on whether AI agents can assess analytical robustness: whether a conclusion holds across reasonable analytical choices, given the same claim and data. In an example from Wilde et al. (2010), four human reanalyses of one claim support it, find it inconclusive or reach the opposite conclusion, and Aczel et al. found 34% of claims fully robust across independent reanalyses. As multi-analyst studies are hard to scale, PaperRobust takes a paper, a focal claim and the original data, generates candidate methods, screens them for quality, relevance and redundancy, runs the accepted ones and reports how conclusions vary. First results on 60 papers compare 306 PaperRobust analyses with 306 human reanalyses: 75.2% versus 71.6% support the claim, and majority conclusions match for 50 papers (83.3%). Speaker: Qian Ma (Penn State University) Authors: Qian Ma, Bang Nguyen, Isham Kalappurackal Mansoor, Dominik Soós, Timothy M. Errington, Shakhlo Nematova, Jian Wu, Meng Jiang and Sarah Rajtmajer Talk title: PaperRobust: a multi-agent framework for assessing analytical robustness Session: Session 8 · Does the finding survive another analyst? Recorded at AI in Metascience 2026, an online conference organised by FORRT (Framework for Open and Reproducible Research Training), 28–30 September 2026. Day 3, Wednesday 30 September: Workshops, and analyst variability. Programme and abstracts: https://forrt.org/ai-metascience/ FORRT: https://forrt.org