The Metascience Observatory: an AI-built replication database and forensic metascience | Dan Elton

Dan Elton introduces the Metascience Observatory, a non-profit building an AI-assisted database of replications across all fields of science. Elton sets out a taxonomy running from direct and close replications to conceptual ones, and a pipeline of keyword searches, full-text conversion and LLM extraction. A validation against FORRT's FReD data found 94 per cent accuracy in identifying original studies. Analyses of the database show replication rates stable across publication years, weakly related to impact factor or journal rank, and similar citation counts for about 4,500 papers that passed or failed replication. The data will merge into FORRT's FReD and FLoRA datasets. Finally, Elton presents the Forensic Metascience Toolkit, open-source modules for image analysis, tortured phrases, statistical checks and peer review, tested on Paul Litvak's benchmark. https://metascienceobservatory.org/ Speaker: Dan Elton (The Metascience Observatory) Talk title: The Metascience Observatory: an AI-built replication database and experiments with AI for forensic metascience Session: Session 7 · Machines that build and test the replication record Recorded at AI in Metascience 2026, an online conference organised by FORRT (Framework for Open and Reproducible Research Training), 28–30 September 2026. Day 2, Tuesday 29 September: Replications and reproductions. Programme and abstracts: https://forrt.org/ai-metascience/ FORRT: https://forrt.org