Enhancing systematic literature reviews with generative artificial intelligence: development, applications, and performance evaluation
Authors:
Top Cited Papers
Open Access
- 27 February 2025
- journal article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 32 (4) , 616-625
Abstract
Objectives: We developed and validated a large language model (LLM)-assisted system for conducting systematic literature reviews in health technology assessment (HTA) submissions. Materials and Methods: We developed a five-module system using abstracts acquired from PubMed: (1) literature search query setup; (2) study protocol setup using population, intervention/comparison, outcome, and study type (PICOs) criteria; (3) LLM-assisted abstract screening; (4) LLM-assisted data extraction; and (5) data summarization. The system incorporates a human-in-the-loop design, allowing real-time PICOs criteria adjustment. This is achieved by collecting information on disagreements between the LLM and human reviewers regarding inclusion/exclusion decisions and their rationales, enabling informed PICOs refinement. We generated four evaluation sets including relapsed and refractory multiple myeloma (RRMM) and advanced melanoma to evaluate the LLM's performance in three key areas: (1) recommending inclusion/exclusion decisions during abstract screening, (2) providing valid rationales for abstract exclusion, and (3) extracting relevant information from included abstracts. Results: The system demonstrated relatively high performance across all evaluation sets. For abstract screening, it achieved an average sensitivity of 90%, F1 score of 82, accuracy of 89%, and Cohen's κ of 0.71, indicating substantial agreement between human reviewers and LLM-based results. In identifying specific exclusion rationales, the system attained accuracies of 97% and 84%, and F1 scores of 98 and 89 for RRMM and advanced melanoma, respectively. For data extraction, the system achieved an F1 score of 93. Discussion: Results showed high sensitivity, Cohen's κ, and PABAK for abstract screening, and high F1 scores for data extraction. This human-in-the-loop AI-assisted SLR system demonstrates the potential of GPT-4's in context learning capabilities by eliminating the need for manually annotated training data. In addition, this LLM-based system offers subject matter experts greater control through prompt adjustment and real-time feedback, enabling iterative refinement of PICOs criteria based on performance metrics. Conclusion: The system demonstrates potential to streamline systematic literature reviews, potentially reducing time, cost, and human errors while enhancing evidence generation for HTA submissions.Keywords
This publication has 34 references indexed in Scilit:
- Screening articles for systematic reviews with ChatGPTJournal of Computer Languages, 2024
- Can large language models replace humans in systematic reviews? Evaluating GPT‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languagesResearch Synthesis Methods, 2024
- Living Health Technology Assessment: Issues, Challenges and OpportunitiesPharmacoEconomics, 2023
- A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviewsDecision Analytics Journal, 2023
- In‐depth evaluation of machine learning methods for semi‐automating article screening in a systematic review of mechanistic literatureResearch Synthesis Methods, 2022
- Using artificial intelligence methods for systematic review in health sciences: A systematic reviewResearch Synthesis Methods, 2022
- SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimationEnvironment International, 2020
- The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trialsContemporary Clinical Trials Communications, 2019
- Best Practice Guidelines for Abstract Screening Large‐Evidence Systematic Reviews and Meta‐AnalysesResearch Synthesis Methods, 2019
- Bias, prevalence and kappaJournal of Clinical Epidemiology, 1993