Don't Stop Me Now, `Cause I'm Having a Good Time Screening: Evaluation of Stopping Methods for Safe Use of Priority Screening in Systematic Reviews
Open Access
- 21 January 2026
- journal article
- review article
- Published by Wiley in Cochrane Evidence Synthesis and Methods
- Vol. 4 (1)
Abstract
Introduction: Priority screening has the potential to reduce the number of records that need to be annotated in systematic literature reviews. So‐called technology‐assisted reviews (TAR) use machine‐learning with prior include/exclude annotations to continuously rank unseen records by their predicted relevance to find relevant records earlier. In this article, we present a systematic evaluation of methods to determine when it is safe to stop screening when using prioritization. Methods: We implement an open‐source evaluation framework that features a novel method to generate rankings and simulate priority screening processes for 81 real‐world data sets. We use these simulations to evaluate 15 statistical or rule‐based (heuristic) stopping methods, testing a range of hyperparameters for each. Results: The work‐saving potential and performance of stopping criteria heavily rely on “good” rankings, which are typically not achieved by a single ranking algorithm across the entire screening process. Our evaluation shows that almost all existing stopping methods either fail to reliably stop without missing relevant records or fail to utilize the full potential work‐savings. Only one method reliably meets the set recall target, but stops conservatively. Conclusions: Many digital evidence synthesis tools provide priority screening features that are already used in many research projects. However, the theoretical work‐savings demonstrated in retrospective simulations of prioritization can only be unlocked with safe and reproducible stopping criteria. Our results highlight the need for improved stopping methods and guidelines on how to responsibly use priority screening. We also urge screening platforms to provide indicators and authors to transparently report metrics when automating (parts of) their synthesis.Keywords
All Related Versions
Funding Information
- European Research Council (22‐Z105‐05/002/001)
This publication has 45 references indexed in Scilit:
- Evaluating the effectiveness of large language models in abstract screening: a comparative analysisSystematic Reviews, 2024
- Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domainSystematic Reviews, 2024
- The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analysesSystematic Reviews, 2024
- Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles?Expert Review of Pharmacoeconomics & Outcomes Research, 2023
- Artificial intelligence in systematic reviews: promising when appropriately usedBMJ Open, 2023
- Papers and patents are becoming less disruptive over timeNature, 2023
- Slowed canonical progress in large fields of scienceProceedings of the National Academy of Sciences, 2021
- Statistical stopping criteria for automated screening in systematic reviewsSystematic Reviews, 2020
- Rayyan—a web and mobile app for systematic reviewsSystematic Reviews, 2016
- Reducing Workload in Systematic Review Preparation Using Automated Citation ClassificationJournal of the American Medical Informatics Association, 2006