Automated identification of incidental hepatic steatosis on Emergency Department imaging using large language models
Authors:
- Tyrus VongTVTyrus Vong
- Nicholas RizerNRNicholas Rizer
- Vedant JainVJVedant Jain
- Valerie L. ThompsonVTValerie L. Thompson
- Mark DredzeMDMark Dredze
- Eili Y. KleinEKEili Y. Klein
- Jeremiah S. HinsonJHJeremiah S. Hinson
- Tanjala PurnellTPTanjala Purnell
- Stephen KwakSKStephen Kwak
- Tinsay WoretaTWTinsay Woreta
- Alexandra T. StraussASAlexandra T. Strauss
Open Access
- 19 February 2025
- journal article
- Vol. 9 (3)
Abstract
Hepatic steatosis is a precursor to more severe liver disease, increasing morbidity and mortality risks. In the Emergency Department, routine abdominal imaging often reveals incidental hepatic steatosis that goes undiagnosed due to the acute nature of encounters. Imaging reports in the electronic health record contain valuable information not easily accessible as discrete data elements. We hypothesized that large language models could reliably detect hepatic steatosis from reports without extensive natural language processing training. We identified 200 adults who had CT abdominal imaging in the Emergency Department between August 1, 2016, and December 31, 2023. Using text from imaging reports and structured prompts, 3 Azure OpenAI models (ChatGPT 3.5, 4, 4o) identified patients with hepatic steatosis. We evaluated model performance regarding accuracy, inter-rater reliability, sensitivity, and specificity compared to physician reviews. The accuracy for the models was 96.2% for v3.5, 98.3% for v4, and 98.8% for v4o. Inter-rater reliability ranged from 0.99 to 1.00 across 10 iterations. Mean model confidence scores were 2.9 (SD 0.8) for v3.5, 3.9 (SD 0.3) for v4, and 4.0 (SD 0.07) for v4o. Incorrect evaluations were 76 (3.8%) for v3.5, 34 (1.7%) for v4, and 25 (1.3%) for v4o. All models showed sensitivity and specificity above 0.9. Large language models can assist in identifying incidental conditions from imaging reports that otherwise may be missed opportunities for early disease intervention. Large language models are a democratization of natural language processing by allowing for a user-friendly, expansive analyses of electronic medical records without requiring the development of complex natural language processing models.Keywords
This publication has 13 references indexed in Scilit:
- Evaluation of GPT-4 ability to identify and generate patient instructions for actionable incidental radiology findingsJournal of the American Medical Informatics Association, 2024
- ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practiceFrontiers in Medicine, 2023
- Prevalence of Steatotic Liver Disease (MASLD, MetALD, and ALD) in the United States: NHANES 2017–2020Clinical Gastroenterology and Hepatology, 2023
- Organizational preparedness for the use of large language models in pathology informaticsJournal of Pathology Informatics, 2023
- Large-scale identification of undiagnosed hepatic steatosis using natural language processingeClinicalMedicine, 2023
- Incidental Radiology Findings on Computed Tomography Studies in Emergency Department Patients: A Systematic Review and Meta-AnalysisAnnals of Emergency Medicine, 2022
- A strategy for validation of variables derived from large-scale electronic health record dataJournal of Biomedical Informatics, 2021
- Complications, morbidity and mortality of nonalcoholic fatty liver diseaseMetabolism, 2020
- Non-invasive diagnosis of hepatic steatosisHepatology International, 2016
- Development of a simple noninvasive index to predict significant fibrosis in patients with HIV/HCV coinfectionHepatology, 2006