MEDBind: Unifying Language and Multimodal Medical Data Embeddings
Preprint
Abstract
Medical vision-language pretraining models (VLPM) have achieved remarkable progress in fusing chest X-rays (CXR) with clinical texts, introducing image-text data binding approaches that enable zero-shot learning and downstream clinical tasks. However, the current landscape lacks the holistic integration of additional medical modalities, such as electrocardiograms (ECG). We present MEDBind (Medical Electronic patient recorD), which learns joint embeddings across CXR, ECG, and medical text. Using text data as the central anchor, MEDBind features tri-modality binding, delivering competitive performance in top-K retrieval, zero-shot, and few-shot benchmarks against established VLPM, and the ability for CXR-to-ECG zero-shot classification and retrieval. This seamless integration is achieved through combination of contrastive loss on modality-text pairs with our proposed contrastive loss function, Edge-Modality Contrastive Loss, fostering a cohesive embedding space for CXR, ECG, and text. Finally, we demonstrate that MEDBind can improve downstream tasks by directly integrating CXR and ECG embeddings into a large-language model for multimodal prompt tuning.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Towards Generalist Biomedical AINEJM AI, 2024
- Health system-scale language models are all-purpose prediction enginesNature, 2023
- MIMIC-IV, a freely accessible electronic health record datasetScientific Data, 2023
- Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing2022
- Can AI Help in Screening Viral and COVID-19 Pneumonia?IEEE Access, 2020
- PTB-XL, a large publicly available electrocardiography datasetScientific Data, 2020
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible PneumoniaRadiology: Artificial Intelligence, 2019
- Mechanisms of physiological and pathological cardiac hypertrophyNature Reviews Cardiology, 2018
- Preparing a collection of radiology examinations for distribution and retrievalJournal of the American Medical Informatics Association, 2015