Large Language Model Demonstrates Human-Comparable Sensitivity in Initial Screening of Systematic Reviews: A Semi-Automated Strategy Using GPT-3.5
Authors:
Preprint
- 27 July 2023
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Background: The screening process for systematic reviews is resource-intensive. Although previous machine learning solutions have reported reductions in workload, there was a critical risk of excluding relevant papers. In this study, we explore the application of large language models (LLMs), specifically the generative pre-trained transformer (GPT)-3.5, to streamline the initial screening process for systematic reviews. We developed a unique GPT-3.5-based screening method and assessed its performance against traditional human screening, with a focus on sensitivity. Methods: We used our two previous systematic review articles related to the treatment of bipolar disorder. For the first article, we engaged in prompt engineering, referencing correct answers derived from secondary screening. Following this, we used the prompts we created to implement screening for the second article. After these processes were performed, the latest (0613) model was released. Ultimately, we applied both the previous (0301) and 0613 GPT-3.5 models to screen the two articles, comparing their assessments to those performed by human evaluators. Findings: The 0301 and 0613 models were able to screen 20 and 50 records per minute, respectively. Both models did not miss any records included in meta-analysis. The sensitivity of the 0301 model and the 0613 model in the first study was 86.7% and 77.4%, respectively, while in the second study, it was 97.9% and 81.3%, respectively. These results were equivalent to those of human evaluators: 86.7 to 100% for the first study and 77.6 to 97.9% for the second study. Interpretation: Our screening method using GPT-3.5 greatly outperformed previous machine learning-based methods and showed sensitivity close to human screening performance. Future studies should endeavor to generalize this methodology and validate its utility in various medical and non-medical settings. Funding: This work was supported by the Japan Society for the Promotion of Science (JSPS) KAKENHI (grant number 22K15778). Declaration of Interest: We declare no competing interests.Keywords
This publication has 18 references indexed in Scilit:
- The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviewsBMC Medical Research Methodology, 2020
- Error rates of human reviewers during abstract screening in systematic reviewsPLOS ONE, 2020
- Efficacy and safety of bright light therapy for manic and depressive symptoms in patients with bipolar disorder: A systematic review and meta‐analysisPsychiatry and Clinical Neurosciences, 2020
- Cochrane Handbook for Systematic Reviews of Interventions2019
- Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR)Systematic Reviews, 2018
- Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning toolSystematic Reviews, 2018
- Can abstract screening workload be reduced using text mining? User experiences of the tool RayyanResearch Synthesis Methods, 2017
- Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewersSystematic Reviews, 2015
- Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviewsResearch Synthesis Methods, 2013
- Scoping studies: towards a methodological frameworkInternational Journal of Social Research Methodology, 2005