Large Language Model in Medical Information Extraction from Titles and Abstracts with Prompt Engineering Strategies: A Comparative Study of GPT-3.5 and GPT-4
Authors:
- Yiyi TangYTYiyi Tang
- Ziyan XiaoZXZiyan Xiao
- Xue LiXLXue Li
- Qiwen FangQFQiwen Fang
- Qingpeng ZhangQZQingpeng Zhang
- Daniel Yee Tak FongDaniel Yee Tak Fong
- Francisco Tsz Tsun LaiFLFrancisco Tsz Tsun Lai
- Celine Sze Ling ChuiCCCeline Sze Ling Chui
- Esther Wai Yin ChanECEsther Wai Yin Chan
- Ian Chi Kei WongIWIan Chi Kei Wong
Preprint
Abstract
Background Large language models (LLMs) have significantly enhanced the Natural Language Processing (NLP), offering significant potential in facilitating medical literature review. However, the accuracy, stability and prompt strategies associated with LLMs in extracting complex medical information have not been adequately investigated. Our study assessed the capabilities of GPT-3.5 and GPT-4.0 in extracting or summarizing seven crucial medical information items from the title and abstract of research papers. We also validated the impact of prompt engineering strategies and the effectiveness of evaluating metrics. Methodology We adopted a stratified sampling method to select 100 papers from the teaching schools and departments in the LKS Faculty of Medicine, University of Hong Kong, published between 2015 and 2023. GPT-3.5 and GPT-4.0 were instructed to extract seven pieces of information, including study design, sample size, data source, patient, intervention, comparison, and outcomes. The experiment incorporated three prompt engineering strategies: persona, chain-of-thought and few-shot prompting. We employed three metrics to assess the alignment between the GPT output and the ground truth: BERTScore, ROUGE-1 and a self-developed GPT-4.0 evaluator. Finally, we evaluated and compared the proportion of correct answers among different GPT versions and prompt engineering strategies. Results GPT demonstrated robust capabilities in accurately extracting medical information from titles and abstracts. The average accuracy of GPT-4.0, when paired with the optimal prompt engineering strategy, ranged from 0.688 to 0.964 among the seven items, with sample size achieving the highest score and intervention yielding the lowest. GPT version was shown to be a statistically significant factor in model performance, but prompt engineering strategies did not exhibit cumulative effects on model performance. Additionally, our results showed that the GPT-4.0 evaluator outperformed the ROUGE-1 and BERTScore in assessing the alignment of information (Accuracy: GPT-4.0 Evaluator: 0.9714, ROUGE-1: 0.9429, BERTScore: 0.8714). Conclusion Our result confirms the effectiveness of LLMs in extracting medical information, suggesting their potential as efficient tools for literature review. We recommend utilizing an advanced version of LLMs to enhance the model performance, while prompt engineering strategies should be tailored to the specific tasks. Additionally, LLMs show promise as an evaluation tool to assess the model performance related to complex information processing.Keywords
This publication has 26 references indexed in Scilit:
- ChatGPT in healthcare: A taxonomy and systematic reviewComputer Methods and Programs in Biomedicine, 2024
- Methods for using Bing's AI‐powered search engine for data extraction for a systematic reviewResearch Synthesis Methods, 2023
- The impact of prompt engineering in large language model performance: a psychiatric exampleJournal of Medical Artificial Intelligence, 2023
- Creation and Adoption of Large Language Models in MedicineJAMA, 2023
- Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google BardEBioMedicine, 2023
- Application ChatGPT in conducting systematic reviews and meta-analysesBritish Dental Journal, 2023
- Large Language Model Demonstrates Human-Comparable Sensitivity in Initial Screening of Systematic Reviews: A Semi-Automated Strategy Using GPT-3.5SSRN Electronic Journal, 2023
- Large language model AI chatbots require approval as medical devicesNature Medicine, 2023
- Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviewsSystematic Reviews, 2016
- Using text mining for study identification in systematic reviews: a systematic review of current approachesSystematic Reviews, 2015