Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study
Authors:
- Anoeska SchipperASAnoeska Schipper
- Peter BelgersPBPeter Belgers
- Rory David O'COnnorRORory David O'COnnor
- Lieke van de WouwLWLieke van de Wouw
- Luc BuiltjesLBLuc Builtjes
- Joeran S BosmaJBJoeran S Bosma
- Ron KustersRKRon Kusters
- Steef KurstjensSKSteef Kurstjens
- Matthieu RuttenMRMatthieu Rutten
- Bram van GinnekenBGBram van Ginneken
Top Cited Papers
Open Access
- 30 April 2026
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 14 (1) , e81500
Abstract
Background: Most clinically relevant information in emergency department (ED) visits is documented in free text, limiting reuse for research and clinical decision support. Despite growing interest in large language model (LLM)–based feature extraction, very few studies have examined it directly on ED reports. Existing work has mainly addressed binary tasks and rarely evaluated their impact on downstream prediction models. Furthermore, evidence for small multilingual LLMs remains limited, especially for underrepresented languages such as Dutch. Locally deployable LLMs could enable automated feature extraction for decision support systems without increasing physician workload. Objective: We aim to evaluate whether a small open-source LLM (Qwen 2.5:14B) can automatically extract 16 clinical signs and symptoms from ED reports and use these as input for an appendicitis prediction model. LLM performance under minimal and optimized 0-shot prompts was assessed against researcher annotations (reference standard) and physician annotations. Methods: This retrospective study used 336 ED reports from patients presenting with acute abdominal pain to a Dutch teaching hospital (2016-2023). One hundred reports were randomly selected to develop a minimal and an optimized 0-shot prompt strategy. The remaining 236 reports, reserved for evaluation, were annotated by 2 ED physicians and processed by the LLM to extract 16 signs and symptoms, covering binary, multiclass, and multilabel classification tasks. These features were used as input to the HIVE (History, Intake, Vitals, Examination) appendicitis prediction model. LLM extraction accuracy, sensitivity, and specificity were measured against the researcher’s (reference standard) and physician annotations. The HIVE model’s area under the receiver operating characteristic curve was evaluated using LLM-extracted vs physician-annotated features. Results: Among 336 ED reports from patients with acute abdominal pain (median age 41, IQR 22‐62 years, 205/336, 61% female), 50% (167/336) had appendicitis. The LLM achieved weighted average accuracies of 0.910 (95% CI (0.018) with minimal prompts and 0.929 (95% CI ±0.016) with optimized prompts, vs 0.961 (95% CI ±0.012) and 0.951 (95% CI ±0.015) for physicians. Corresponding HIVE model area under the receiver operating characteristic curves were 0.871 (95% CI ±0.019) and 0.911 (95% CI ±0.014) with LLM inputs under the minimal and optimized prompts, compared to 0.917 (95% CI ±0.015) and 0.924 (95% CI ±0.018) for physician inputs. Conclusions: A small locally deployable multilingual LLM can approach physician-level accuracy in extracting structured binary, multiclass, and multilabel clinical data from free-text Dutch ED reports, while preserving patient privacy, interpretability, and statistical transparency for downstream diagnostic modeling.Keywords
This publication has 34 references indexed in Scilit:
- Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: A data-driven approach for improved classificationMedical Image Analysis, 2024
- How artificial intelligence could transform emergency careThe American Journal of Emergency Medicine, 2024
- Deep learning-based natural language processing for detecting medical symptoms and histories in emergency patient triageThe American Journal of Emergency Medicine, 2023
- Predicting the Survival of Patients With Cancer From Their Initial Oncology Consultation Document Using Natural Language ProcessingJAMA Network Open, 2023
- Clinical narrative-aware deep neural network for emergency department critical outcome predictionJournal of Biomedical Informatics, 2023
- Machine learning for predicting readmission risk among the frail: Explainable AI for healthcarePatterns, 2021
- Reducing the negative appendectomy rate with the laparoscopic appendicitis score; a multicenter prospective cohort and validation studyInternational Journal of Surgery, 2020
- Factors Associated With Potentially Missed Diagnosis of Appendicitis in the Emergency DepartmentJAMA Network Open, 2020
- Utilizing uncoded consultation notes from electronic medical records for predictive modeling of colorectal cancerArtificial Intelligence in Medicine, 2016
- Improvement in the Diagnosis of AppendicitisAdvances in Surgery, 2013