Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (25) , 26796-26804
Abstract
The performance of multimodal models often deteriorates when modality absence occurs. The absence disrupts the learned inter-modal correlations, resulting in biased multimodal representations. This challenge is especially pronounced when the absence is pervasive, affecting both the training and inference phases. Recent studies have attempted to reconstruct the missing information; however, most of them require complete supervision, which is seldom available in scenarios of pervasive absence. The quality of reconstruction remains a critical issue. Alternatively, others aim to learn robust representations from the available modalities but the substantial variations and biases are not fully addressed. This paper introduces the Multimodal Generalization and Refinement (MGR) framework to mitigate the issue of pervasive modality absence. MGR begins by acquiring generalized multimodal representations and iteratively refines them to recognize and calibrate the biased representations. Initially, multimodal samples with absence are embedded through foundation models, and MGR integrates independent unimodal features to further enhance generalization. Additionally, a novel mixed-context prompt is adopted to identify biases in both features and correlations. A redistribution operation can then refine these biases through graph pooling, culminating in robust and calibrated multimodal representations, which are suitable for downstream tasks. Comprehensive experiments on four benchmark datasets demonstrate that the proposed MGR framework outperforms state-of-the-art methods, effectively mitigating the impact of pervasive modality absence.Keywords
This publication has 12 references indexed in Scilit:
- MaskMentor: Unlocking the Potential of Masked Self-Teaching for Missing Modality RGB-D Semantic Segmentation2024
- Similar modality completion-based multimodal sentiment analysis under uncertain missing modalitiesInformation Fusion, 2024
- Towards Robust Multimodal Prompting with Missing Modalities2024
- Towards Robust Multimodal Sentiment Analysis Under Uncertain Signal MissingIEEE Signal Processing Letters, 2023
- Modality translation-based multimodal sentiment analysis under uncertain missing modalitiesInformation Fusion, 2023
- Style transfer in conditional GANs for cross-modality synthesis of brain magnetic resonance imagesComputers in Biology and Medicine, 2022
- AF: An Association-Based Fusion Method for Multi-Modal ClassificationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Feature-level fusion approaches based on multimodal EEG data for depression recognitionInformation Fusion, 2020
- Multimodal Machine Learning: A Survey and TaxonomyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- VIGAN: Missing view imputation with generative adversarial networks2017