Dynamic Memory-Augmented Retrieval and Synthesis for Robust Incomplete Multimodal Learning

Authors:
  • ZWZhengjue Wang
    Zhengjue Wang
  • HZHao Zhang
    Hao Zhang
  • JSJianqiao Sun
    Jianqiao Sun
  • JMJiawei Ma
    Jiawei Ma
  • BCBo Chen
    Bo Chen

Abstract
Multimodal learning, integrating data from multiple sources, has demonstrated significant potential for enhancing the performance of downstream tasks. However, a common challenge in real-world applications is the missing-modality cases, also known as the incomplete multimodal learning problem where data from one or more sources may be unavailable during training or inference, due to factors like sensor failure, privacy constraints or other factors. Conventional methods frequently discard incomplete samples, which significantly limits the model's generalization capability and induces potential overfitting. To overcome these limitations, we propose an incomplete multimodal learning method, named Dynamic Memory-Augmented Retrieval and Synthesis (DynaMARS). Our model is composed of modality-specific encoders with dynamic memory banks and a specialized Memory-Guided Fusion Module (MGFM). Specifically, DynaMARS leverages the memory banks to retrieve rich historical information for missing modalities, and then the retrieved features are synthesized with available modality features to construct a robust and comprehensive multimodal representation. The entire model is trained in an end-to-end manner using a multi-task and supervised optimization paradigm. Extensive experiments across four benchmark datasets have demonstrated that DynaMARS delivers state-of-the-art or comparable performances in handling various missing modality patterns. And it exhibits superior robustness particularly in scenarios with severe modality missingness.

This publication has 15 references indexed in Scilit: