COMPRER: A Multimodal Multi-Objective Pretraining Framework for Enhanced Medical Image Representation
Preprint
Abstract
Substantial advances in multi-modal Artificial Intelligence (AI) facilitate the combination of diverse medical modalities to achieve holistic health assessments. We present COMPRER, a novel multi-modal, multi-objective pretraining framework which enhances medical-image representation, diagnostic inferences, and prognosis of diseases. COMPRER employs a multi-objective training framework, where each objective introduces distinct knowledge to the model. This includes a multi-modal loss that consolidates information across different imaging modalities; A temporal loss that imparts the ability to discern patterns over time; Medical-measure prediction adds appropriate medical insights; Lastly, reconstruction loss ensures the integrity of image structure within the latent space. Despite the concern that multiple objectives could weaken task performance, our findings show that this combination actually boosts outcomes on certain tasks. Here, we apply this framework to both fundus images and carotid ultrasound, and validate our downstream tasks capabilities by predicting both current and future cardiovascular conditions. COMPRER achieved higher Area Under the Curve (AUC) scores in evaluating medical conditions compared to existing models on held-out data. On the Out-of-distribution (OOD) UK-Biobank dataset COMPRER maintains favorable performance over well-established models with more parameters, even though these models were trained on 75× more data than COMPRER. In addition, to better assess our model’s performance in contrastive learning, we introduce a novel evaluation metric, providing deeper understanding of the effectiveness of the latent space pairing.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- A foundation model for generalizable disease detection from retinal imagesNature, 2023
- Cross-modal autoencoder framework learns holistic representations of cardiovascular stateNature Communications, 2023
- Foundation models for generalist medical artificial intelligenceNature, 2023
- Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learningNature Biomedical Engineering, 2022
- A deep learning system for detecting diabetic retinopathy across the disease spectrumNature Communications, 2021
- 10 K: a large‐scale prospective longitudinal study in IsraelEuropean Journal of Epidemiology, 2021
- Artificial intelligence-enhanced electrocardiography in cardiovascular disease managementNature Reviews Cardiology, 2021
- Identifying pneumonia in chest X-rays: A deep learning approachMeasurement, 2019
- Prediction of cardiovascular risk factors from retinal fundus photographs via deep learningNature Biomedical Engineering, 2018
- Technology Insight: ultrasound measurement of carotid plaque—patient management, genetic research, and therapy evaluationNature Clinical Practice Neurology, 2006