Robust Incomplete-Modality Alignment for Ophthalmic Disease Grading and Diagnosis via Labeled Optimal Transport
Preprint
Abstract
Multimodal ophthalmic imaging-based diagnosis integrates color fundus image with optical coherence tomography (OCT) to provide a comprehensive view of ocular pathologies. However, the uneven global distribution of healthcare resources often results in real-world clinical scenarios encountering incomplete multimodal data, which significantly compromises diagnostic accuracy. Existing commonly used pipelines, such as modality imputation and distillation methods, face notable limitations: 1)Imputation methods struggle with accurately reconstructing key lesion features, since OCT lesions are localized, while fundus images vary in style. 2)distillation methods rely heavily on fully paired multimodal training data. To address these challenges, we propose a novel multimodal alignment and fusion framework capable of robustly handling missing modalities in the task of ophthalmic diagnostics. By considering the distinctive feature characteristics of OCT and fundus images, we emphasize the alignment of semantic features within the same category and explicitly learn soft matching between modalities, allowing the missing modality to utilize existing modality information, achieving robust cross-modal feature alignment under the missing modality. Specifically, we leverage the Optimal Transport for multi-scale modality feature alignment: class-wise alignment through predicted class prototypes and feature-wise alignment via cross-modal shared feature transport. Furthermore, we propose an asymmetric fusion strategy that effectively exploits the distinct characteristics of OCT and fundus modalities. Extensive evaluations on three large ophthalmic multimodal datasets demonstrate our model's superior performance under various modality-incomplete scenarios, achieving Sota performance in both complete modality and inter-modality incompleteness conditions. Code is available at https://github.com/Qinkaiyu/RIMAKeywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Confidence-aware multi-modality learning for eye disease screeningMedical Image Analysis, 2024
- Similar modality completion-based multimodal sentiment analysis under uncertain missing modalitiesInformation Fusion, 2024
- GAMMA challenge: Glaucoma grAding from Multi-Modality imAgesMedical Image Analysis, 2023
- Tuning Multi-mode Token-level Prompt Alignment across Modalities2023
- CASF-Net: Cross-attention and cross-scale fusion network for medical image segmentationComputer Methods and Programs in Biomedicine, 2022
- OTKGE: Multi-Modal Knowledge Graph Embeddings Via Optimal Transport2022
- Convolutional Network With Twofold Feature Augmentation for Diabetic Retinopathy Recognition From Multi-Modal ImagesIEEE Journal of Biomedical and Health Informatics, 2020
- The Development, Commercialization, and Impact of Optical Coherence TomographyInvestigative Ophthalmology & Visual Science, 2016
- A review of optical coherence tomography angiography (OCTA)International Journal of Retina and Vitreous, 2015
- Concerning nonnegative matrices and doubly stochastic matricesPacific Journal of Mathematics, 1967