Leveraging Large Language Models for Accurate Retrieval of Patient Information From Medical Reports: Systematic Evaluation Study (Preprint)
Preprint
- 14 November 2024
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND The digital transformation of health care has introduced both opportunities and challenges, particularly in managing and analyzing the vast amounts of unstructured medical data generated daily. There is a need to explore the feasibility of generative solutions in extracting data from medical reports, categorized by specific criteria. OBJECTIVE This study aimed to investigate the application of large language models (LLMs) for the automated extraction of structured information from unstructured medical reports, using the LangChain framework in Python. METHODS Through a systematic evaluation of leading LLMs—GPT-4o, Llama 3, Llama 3.1, Gemma 2, Qwen 2, and Qwen 2.5—using zero-shot prompting techniques and embedding results into a vector database, this study assessed the performance of LLMs in extracting patient demographics, diagnostic details, and pharmacological data. RESULTS Evaluation metrics, including accuracy, precision, recall, and F1-score, revealed high efficacy across most categories, with GPT-4o achieving the highest overall performance (91.4% accuracy). CONCLUSIONS The findings highlight notable differences in precision and recall between models, particularly in extracting names and age-related information. There were challenges in processing unstructured medical text, including variability in model performance across data types. Our findings demonstrate the feasibility of integrating LLMs into health care workflows; LLMs offer substantial improvements in data accessibility and support clinical decision-making processes. In addition, the paper describes the role of retrieval-augmented generation techniques in enhancing information retrieval accuracy, addressing issues such as hallucinations and outdated data in LLM outputs. Future work should explore the need for optimization through larger and more diverse training datasets, advanced prompting strategies, and the integration of domain-specific knowledge to improve model generalizability and precision.Keywords
All Related Versions
- Published version: JMIR AI, 4 (1), e68776.
This publication has 14 references indexed in Scilit:
- Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluationnpj Mental Health Research, 2024
- Hallucinations in Large Multilingual Translation ModelsTransactions of the Association for Computational Linguistics, 2023
- Can ChatGPT pass the life support exams without entering the American heart association course?Resuscitation, 2023
- Enabling Large Language Models to Generate Text with Citations2023
- Automated Identification and Measurement Extraction of Pancreatic Cystic Lesions from Free-Text Radiology Reports Using Natural Language ProcessingRadiology: Artificial Intelligence, 2022
- Evolution of Smart Healthcare: Telemedicine During COVID-19 PandemicJournal of The Institution of Engineers (India): Series B, 2021
- Using a knowledge graph data structure to analyze text documents (VAST challenge 2014 MC1)2014
- Computational Medicine in Data Mining and Modeling2013
- Information Extraction from Text2012
- Telemedicine Can Make Healthcare GreenerTelemedicine and e-Health, 2010