Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation
Authors:
Open Access
- 10 May 2025
- journal article
- review article
- Published by Springer Nature in BMC Medical Research Methodology
- Vol. 25 (1) , 1-9
Abstract
Systematic reviews (SRs) are essential to formulate evidence-based guidelines but require time-consuming and costly literature screening. Large Language Models (LLMs) can be a powerful tool to expedite SRs. We conducted a comparative study to evaluate the performance of a commercial tool, Rayyan, and an in-house LLM-based system in automating the screening of a completed SR on Vitamin D and falls. The SR retrieved 14,439 articles, and Rayyan was trained with 2,000 manually screened articles to categorize the rest as most likely to exclude/include, likely to exclude/include and undecided. We analyzed Rayyan’s title/abstract screening performance using different inclusion thresholds. For the LLM, we used prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening. We evaluated performance using article exclusion rate (AER), false negative rate (FNR), specificity, positive predictive value (PPV), and negative predictive value (NPV). Additionally, we compared the time required to complete screening steps of the SR using both approaches against the manual screening method. Using Rayyan, including considered as undecided or likely to include for title/abstract screening resulted in an AER of 72.1% and an FNR of 5%. The total estimated screening time, including manual review of articles flagged by Rayyan, was 54.7 hours. Lowering the Rayyan threshold to ‘likely to exclude’ reduced the FNR to 0% and the AER to 50.7%, but increased the screening time to 81.3 h. Using the LLM system, after title/abstract and full-text screening, 78 articles remained for manual review, including all 20 identified by traditional methods. The LLM achieved an AER of 99.5%, specificity of 99.6%, PPV of 25.6%, and NPV of 100%, with a total screening time of 25.5 h, including manual review of the 78 articles, reducing the manual screening time by 95.5%. The LLM-based system significantly enhances SR efficiency, compared to manual methods and Rayyan while maintaining low FNR.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Performance of a Large Language Model in Screening CitationsJAMA Network Open, 2024
- Pilot study on large language models for risk-of-bias assessments in systematic reviews: A(I) new type of bias?BMJ Evidence-Based Medicine, 2024
- Can large language models replace humans in systematic reviews? Evaluating GPT‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languagesResearch Synthesis Methods, 2024
- Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessmentBMJ Evidence-Based Medicine, 2024
- Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language ModelsPharmacoEconomics - Open, 2024
- Sensitivity and specificity of alternative screening methods for systematic reviews using text mining toolsJournal of Clinical Epidemiology, 2023
- An open source machine learning framework for efficient and transparent systematic reviewsNature Machine Intelligence, 2021
- Using machine learning to advance synthesis and use of conservation and environmental evidenceConservation Biology, 2018
- Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registryBMJ Open, 2017
- Growth rates of modern science: A bibliometric analysis based on the number of publications and cited referencesJournal of the Association for Information Science and Technology, 2015