Knowledge-Augmented Large Language Model for Multimodal Electronic Health Record–Based Risk Prediction: Development and Validation Study (Preprint)
Preprint
- 24 November 2025
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Accurate clinical outcome prediction using electronic health records (EHRs) is crucial for patient care and resource allocation. EHRs include both structured data and rich, unstructured clinical notes. However, prior machine learning methods struggle with the multimodality, long context of notes, and severe class imbalance in clinical tasks. OBJECTIVE This study aimed to introduce and evaluate KAMELEON (Knowledge-Augmented Multimodal EHR Learning for Outcome Prediction), a unified, 2-stage hybrid framework that integrates diverse EHR modalities and external biomedical knowledge to enhance clinical risk prediction. METHODS This study used the publicly available, deidentified Medical Information Mart for Intensive Care-III dataset, which includes structured and unstructured data for over 40,000 intensive care unit patients. The 2 tasks studied were 30-day readmission (403/10,031, 4% positive rate) and in-hospital mortality prediction (2423/17,903, 13% positive rate). Train-test splits were patient-disjoint (80:20). Performance was evaluated against general and medical large language models (LLMs) and structured baselines. Key metrics included the area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), and macro F1-score. RESULTS The KAMELEON framework consistently outperformed all existing baselines. 30-day readmission: the KAMELEON-balanced random forests model achieved an AUROC of 0.85 and a sensitivity (recall) of 0.79. Ablation analysis shows the critical role of the LLM-generated reasoning, with its removal causing the AUROC to drop from 0.85 to 0.7 and sensitivity to fall by over 80%. In-hospital mortality: the KAMELEON-extreme gradient boosting model achieved an AUROC of 0.92 and an AUPRC of 0.650. Unstructured-only models showed limited ability to discern mortality, with AUROC values near chance (around 0.51-0.53). CONCLUSIONS To our knowledge, KAMELEON represents one of the first systematic frameworks to enhance LLMs for health care prediction through graph-guided knowledge retrieval combined with structured machine learning. The framework demonstrates superior performance across both prediction tasks, highlighting the synergistic value of combining diverse data modalities and LLM reasoning for robust clinical risk estimation.Keywords
All Related Versions
- Published version: JMIR AI, 5, e88356.
This publication has 39 references indexed in Scilit:
- TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health recordsNature Communications, 2023
- EHR-KnowGen: Knowledge-enhanced multimodal learning for disease diagnosis generationInformation Fusion, 2023
- Association of Inclusion of Medicare Advantage Patients in Hospitals’ Risk-Standardized Readmission Rates, Performance, and Penalty StatusJAMA Network Open, 2021
- Development of Electronic Health Record–Based Prediction Models for 30-Day Readmission Risk Among Patients Hospitalized for Acute Myocardial InfarctionJAMA Network Open, 2021
- Prospective and External Evaluation of a Machine Learning Model to Predict In-Hospital Mortality of Adults at Time of AdmissionJAMA Network Open, 2020
- An empirical evaluation of deep learning for ICD-9 code assignment using MIMIC-III clinical notesComputer Methods and Programs in Biomedicine, 2019
- Assessment of Machine Learning vs Standard Prediction Rules for Predicting Hospital ReadmissionsJAMA Network Open, 2019
- Benchmarking deep learning models on large healthcare datasetsJournal of Biomedical Informatics, 2018
- Will Medicare Readmission Penalties Motivate Hospitals to Reduce Arthroplasty Readmissions?The Journal of Arthroplasty, 2017
- Risk Prediction Models for Hospital ReadmissionJAMA, 2011