Multimodal Fusion of Echocardiogram Images and Electronic Medical Records for Heart Disease Screening: Retrospective Algorithm Development and Validation Study
Open Access
- 14 May 2026
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 14 (1) , e78949
Abstract
Background: Echocardiography is a fundamental imaging modality for the diagnosis of heart disease (HD), but its interpretation remains operator-dependent and lacks standardized, data-driven decision support. Although artificial intelligence has improved image-based diagnosis, the added value and interpretability of integrating routinely collected electronic medical records (EMRs) with echocardiogram (ECHO) for large-scale screening remain underexplored. Objective: This study aims to develop an explainable artificial intelligence framework that integrates multimodal data, including ECHO images and EMRs, with derived temporal and clinical features, to enhance the screening and interpretability of heart disease diagnosis. Methods: In this retrospective single-center study, we analyzed 26,936 patients with linked ECHO and EMR records (1470 with clinically confirmed HD and 25,466 without). We constructed cross-sectional EMR features (demographics, comorbidities, and registration history) and extracted view-specific image features with a simplified Inception-v3 backbone. Five modality-specific feature vectors (4 imaging views + EMR) were concatenated and input to an extreme gradient boosting classifier within a patient-level stratified 5-fold cross-validation framework. To mitigate class imbalance during training, non-HD cases in the training folds were randomly downsampled to a 1:3 HD:non-HD ratio; held-out test folds preserved the original prevalence. Post hoc explainability was provided via Grad-CAM heatmaps for images and Shapley additive explanations analysis for EMR features. Results: For the primary binary task (presence vs absence of HD), the multimodal model achieved an AUC of 0.8147 (SD 0.009) on held-out test folds, compared with 0.7785 (SD 0.014) for the image-only baseline, and 0.7343 (SD 0.009) for the EMR-only baseline. At the clinically selected decision threshold, sensitivity, specificity, positive predictive value, and negative predictive value were 84.6% (SD 1.6%), 72.4% (SD 2%), 15% (SD 2.3%), 98.8% (SD 0.3%) for the multimodal model, versus 82.3% (SD 1.9%), 73.5% (SD 2%), 15.2% (SD 2.1%), 98.6% (SD 0.3%) (image-only), and 77.5% (SD 2.1%), 67% (SD 2.3%), 11.9% (SD 1.7%), 98.1% (SD 0.4%) (EMR-only). Grad-CAM visualizations qualitatively highlighted anatomically and physiologically meaningful regions (eg, valvular structures and abnormal flow jets), while Shapley additive explanations analysis identified age, registration years, sex, and hypertension among the top EMR contributors—findings that align with established cardiovascular risk factors. Conclusions: Integrating echocardiographic images and EMR data in an explainable multimodal framework yields improved and clinically plausible screening performance for heart disease. Future work should focus on external multicenter validation, quantitative assessment of visual explanations against expert annotations, and prospective studies assessing clinical use and workflow integration.Keywords
This publication has 39 references indexed in Scilit:
- Automatic Cardiac Pathology Recognition in Echocardiography Images using Higher Order Dynamic Mode Decomposition and a Vision Transformer for Small DatasetsExpert Systems with Applications, 2024
- Early detection and prediction of Heart Disease using Wearable devices and Deep Learning algorithmsMultimedia Tools and Applications, 2024
- CarpNet : Transformer for mitral valve disease classification in echocardiographic videosInternational Journal of Imaging Systems and Technology, 2023
- Abbreviated Echocardiographic Screening for Rheumatic Heart Disease by Nonexperts with and without Offsite Expert Review: A Diagnostic Accuracy StudyJournal of the American Society of Echocardiography, 2023
- Recent developments in modeling, imaging, and monitoring of cardiovascular diseases using machine learningBiophysical Reviews, 2023
- Recommendations for the Use of Echocardiography in the Evaluation of Rheumatic Heart Disease: A Report from the American Society of EchocardiographyJournal of the American Society of Echocardiography, 2022
- Heart disease diagnosis using deep learning and cardiac color doppler ultrasoundSoft Computing, 2022
- Application of machine learning in screening for congenital heart diseases using fetal echocardiographyThe International Journal of Cardiovascular Imaging, 2022
- Performance of the American Heart Association/American College of Cardiology Pooled Cohort Equations to Estimate Atherosclerotic Cardiovascular Disease Risk by Self-reported Physical Activity LevelsJAMA Cardiology, 2021
- Artificial intelligence, fetal echocardiography, and congenital heart diseasePrenatal Diagnosis, 2021