Unified Multi-Modal Diagnostic Framework With Reconstruction Pre-Training and Heterogeneity-Combat Tuning
- 2 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Biomedical and Health Informatics
- Vol. 29 (5) , 3171-3183
Abstract
Medical multi-modal pre-training has revealed promise in computer-aided diagnosis by leveraging large-scale unlabeled datasets. However, existing methods based on masked autoencoders mainly rely on data-level reconstruction tasks, but lack high-level semantic information. Furthermore, two significant heterogeneity challenges hinder the transfer of pre-trained knowledge to downstream tasks, i.e. , the distribution heterogeneity between pre-training data and downstream data, and the modality heterogeneity within downstream data. To address these challenges, we propose a Unified Medical Multi-modal Diagnostic (UMD) framework with tailored pre-training and downstream tuning strategies. Specifically, to enhance the representation abilities of vision and language encoders, we propose the Multi-level Reconstruction Pre-training (MR-Pretrain) strategy, including a feature-level and data-level reconstruction, which guides models to capture the semantic information from masked inputs of different modalities. Moreover, to tackle two kinds of heterogeneities during the downstream tuning, we present the heterogeneity-combat downstream tuning strategy, which consists of a Task-oriented Distribution Calibration (TD-Calib) and a Gradient-guided Modality Coordination (GM-Coord). In particular, TD-Calib fine-tunes the pre-trained model regarding the distribution of downstream datasets, and GM-Coord adjusts the gradient weights according to the dynamic optimization status of different modalities. Extensive experiments on five public medical datasets demonstrate the effectiveness of our UMD framework, which remarkably outperforms existing approaches on three kinds of downstream tasks.Keywords
All Related Versions
Funding Information
- InnoHK Program
This publication has 40 references indexed in Scilit:
- Towards overcoming data scarcity in materials science: unifying models and datasets with a mixture of experts frameworknpj Computational Materials, 2022
- Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training2022
- GaitForeMer: Self-supervised Pre-training of Transformers via Human Motion Forecasting for Few-Shot Gait Impairment Severity Estimation2022
- Recent advances and clinical applications of deep learning in medical image analysisMedical Image Analysis, 2022
- Self-supervised multimodal reconstruction pre-training for retinal computer-aided diagnosisExpert Systems with Applications, 2021
- Annotation-efficient deep learning for automatic medical image segmentationNature Communications, 2021
- Contrastive Pre-training and Representation Distillation for Medical Visual Question Answering Based on Radiology Images2021
- A survey on incorporating domain knowledge into deep learning for medical image analysisMedical Image Analysis, 2021
- Overcoming Data Limitation in Medical Visual Question Answering2019
- Radiology Objects in COntext (ROCO): A Multimodal Image Dataset2018