Large Language Model Demonstrates Human-Comparable Sensitivity in Initial Screening of Systematic Reviews: A Semi-Automated Strategy Using GPT-3.5

Authors:

Abstract
Background: The screening process for systematic reviews is resource-intensive. Although previous machine learning solutions have reported reductions in workload, there was a critical risk of excluding relevant papers. In this study, we explore the application of large language models (LLMs), specifically the generative pre-trained transformer (GPT)-3.5, to streamline the initial screening process for systematic reviews. We developed a unique GPT-3.5-based screening method and assessed its performance against traditional human screening, with a focus on sensitivity. Methods: We used our two previous systematic review articles related to the treatment of bipolar disorder. For the first article, we engaged in prompt engineering, referencing correct answers derived from secondary screening. Following this, we used the prompts we created to implement screening for the second article. After these processes were performed, the latest (0613) model was released. Ultimately, we applied both the previous (0301) and 0613 GPT-3.5 models to screen the two articles, comparing their assessments to those performed by human evaluators. Findings: The 0301 and 0613 models were able to screen 20 and 50 records per minute, respectively. Both models did not miss any records included in meta-analysis. The sensitivity of the 0301 model and the 0613 model in the first study was 86.7% and 77.4%, respectively, while in the second study, it was 97.9% and 81.3%, respectively. These results were equivalent to those of human evaluators: 86.7 to 100% for the first study and 77.6 to 97.9% for the second study. Interpretation: Our screening method using GPT-3.5 greatly outperformed previous machine learning-based methods and showed sensitivity close to human screening performance. Future studies should endeavor to generalize this methodology and validate its utility in various medical and non-medical settings. Funding: This work was supported by the Japan Society for the Promotion of Science (JSPS) KAKENHI (grant number 22K15778). Declaration of Interest: We declare no competing interests.

This publication has 18 references indexed in Scilit: