Dynamic Memory-Augmented Retrieval and Synthesis for Robust Incomplete Multimodal Learning
Authors:
- Chiyu ChenCCChiyu Chen
- Zhengjue WangZWZhengjue Wang
- Hao ZhangHZHao Zhang
- Jianqiao SunJSJianqiao Sun
- Jiawei MaJMJiawei Ma
- Bo ChenBCBo Chen
Preprint
- 10 March 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Multimodal learning, integrating data from multiple sources, has demonstrated significant potential for enhancing the performance of downstream tasks. However, a common challenge in real-world applications is the missing-modality cases, also known as the incomplete multimodal learning problem where data from one or more sources may be unavailable during training or inference, due to factors like sensor failure, privacy constraints or other factors. Conventional methods frequently discard incomplete samples, which significantly limits the model's generalization capability and induces potential overfitting. To overcome these limitations, we propose an incomplete multimodal learning method, named Dynamic Memory-Augmented Retrieval and Synthesis (DynaMARS). Our model is composed of modality-specific encoders with dynamic memory banks and a specialized Memory-Guided Fusion Module (MGFM). Specifically, DynaMARS leverages the memory banks to retrieve rich historical information for missing modalities, and then the retrieved features are synthesized with available modality features to construct a robust and comprehensive multimodal representation. The entire model is trained in an end-to-end manner using a multi-task and supervised optimization paradigm. Extensive experiments across four benchmark datasets have demonstrated that DynaMARS delivers state-of-the-art or comparable performances in handling various missing modality patterns. And it exhibits superior robustness particularly in scenarios with severe modality missingness.Keywords
This publication has 15 references indexed in Scilit:
- A systematic literature review on incomplete multimodal learning: techniques and challengesSystems Science & Control Engineering, 2025
- MSF-Net: Multi-stage fusion network for emotion recognition from multimodal signals in scalable healthcareInformation Fusion, 2025
- Multimodal hypergraph network with contrastive learning for sentiment analysisNeurocomputing, 2025
- Terrain detection and segmentation for autonomous vehicle navigation: A state-of-the-art systematic reviewInformation Fusion, 2024
- Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing ModalitiesIEEE Transactions on Affective Computing, 2024
- FRAMM: Fair ranking with missing modalities for clinical trial site selectionPatterns, 2024
- Privacy-preserving human activity sensing: A surveyHigh-Confidence Computing, 2024
- Efficient Multimodal Transformer With Dual-Level Feature Restoration for Robust Multimodal Sentiment AnalysisIEEE Transactions on Affective Computing, 2023
- Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial NetworksIEEE Transactions on Medical Imaging, 2019
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008