The hunt for the last relevant paper: blending the best of humans and AI
Authors:
- Rens van de SchootRSRens van de Schoot
- Tale EvenhuisTETale Evenhuis
- Peter LombaersPLPeter Lombaers
- Felix WeijdemaFWFelix Weijdema
- Laurens de BruinLBLaurens de Bruin
- Rutger NeelemanRNRutger Neeleman
- Elizabeth GrandfieldEGElizabeth Grandfield
- Marit SijbrandijMSMarit Sijbrandij
- Jelle Jasper TeijemaJTJelle Jasper Teijema
- Elena JalsovecEJElena Jalsovec
- Michiel Pieter BronMBMichiel Pieter Bron
Open Access
- 15 October 2025
- journal article
- Published by Taylor & Francis in European Journal of Psychotraumatology
- Vol. 16 (1) , 2546214
Abstract
Background: The exponential growth of research literature makes it increasingly difficult to identify all relevant studies for systematic reviews and meta-analyses. While traditional search methods are labour-intensive, modern AI-aided approaches have the potential to act as a powerful 'super-assistant' during both the searching and screening phases.Objective: This paper evaluates how a combined, open-source approach - merging traditional and AI-aided search and screening methods - can help identify all relevant literature up to the 'last relevant paper' for a systematic review on post-traumatic stress symptom (PTSS) trajectories after traumatic events.Method: We applied eight search strategies, including database searches, snowballing, full-text retrieval, and semantic search via OpenAlex. All records were screened using a combination of human reviewers, active learning, and large language models (LLMs) for quality control.Results: On top of replicating the original 6,701 search results, we identified an additional 3,822 records using AI-aided methods. The combination of AI tools and human screening led to 126 relevant studies, with each method uncovering papers the others missed. Notably, machine-aided techniques helped find studies with missing keywords, unusual phrasing, or limited indexing. Across all AI-assisted strategies, 10 additional studies were identified, and while the overall yield was modest, these papers were unique and relevant and would likely have been missed using traditional methods.Conclusions: Our findings demonstrate that even when returns are low, AI-aided approaches can meaningfully enhance coverage and offer a scalable path forward when combined with screening prioritisation. A transparent, hybrid workflow where AI serves as a 'super-assistant' can meaningfully extend the reach of systematic reviews and increase the quality of the findings, but is not ready to replace humans fully.Keywords
All Related Versions
Funding Information
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Sociale en Geesteswetenschappen, NWO (406.22.GO.048)
This publication has 66 references indexed in Scilit:
- Rayyan—a web and mobile app for systematic reviewsSystematic Reviews, 2016
- Early PTSD Symptom Trajectories: Persistence, Recovery, and Response to Treatment: Results from the Jerusalem Trauma Outreach and Prevention Study (J-TOPS)PLOS ONE, 2013
- The Impact of Epidemic Violence on the Prevalence of Psychiatric Disorders in Sao Paulo and Rio de Janeiro, BrazilPLOS ONE, 2013
- THE EFFECT OF ENGLISH-LANGUAGE RESTRICTION ON SYSTEMATIC REVIEW-BASED META-ANALYSES: A SYSTEMATIC REVIEW OF EMPIRICAL STUDIESInternational Journal of Technology Assessment in Health Care, 2012
- Classification of trauma and stressor-related disorders in DSM-5Depression and Anxiety, 2011
- Semi-automated screening of biomedical citations for systematic reviewsBMC Bioinformatics, 2010
- Cross-Topic Learning for Work Prioritization in Systematic Review Creation and UpdateJournal of the American Medical Informatics Association, 2009
- For randomized controlled trials, the quality of reports of complementary and alternative medicine was as good as reports of conventional medicineJournal of Clinical Epidemiology, 2005
- On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutionsBiostatistics, 2001
- Language bias in randomised controlled trials published in English and GermanThe Lancet, 1997