A local, open-weights pipeline for semi-automated replication checklists | Philipp Knöpfle

Philipp Knöpfle presents Re:check, a pipeline that pre-fills a replication checklist for computational communication research with an open-weight model on a local server. The checklist has about 30 items across nine thematic areas; free-text fields only ask where information is located, because descriptive items worked far better than subjective ones. The pipeline extracts verbatim evidence from the paper, answers each item from that evidence, then audits the answers and flags low-confidence ones for human review, in 20 to 30 seconds per paper. Against hand coding of 100 articles, it agrees on which items apply 90.9% of the time and on answers 78% of the time: 0.85 for manuscript-resident items but 0.56 for repository-resident items, mostly through false yes answers. A field experiment at the journal Computational Communication Research comes next. Speaker: Philipp Knöpfle (LMU Munich) Authors: Johannes Breuer (CAIS Bochum and University of Duisburg-Essen), Mario Haim and Philipp Knöpfle (LMU Munich), Xinyue Zhao (CAIS Bochum) Talk title: A local, open-weights pipeline for semi-automated replication checklists Session: Session 5 · Reproducing the record Recorded at AI in Metascience 2026, an online conference organised by FORRT (Framework for Open and Reproducible Research Training), 28–30 September 2026. Day 2, Tuesday 29 September: Replications and reproductions. Programme and abstracts: https://forrt.org/ai-metascience/ FORRT: https://forrt.org