Towards Robust Multimodal Prompting with Missing Modalities
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8070-8074
Abstract
Recently, multimodal prompting, which introduces learnable missing-aware prompts for all missing modality cases, has exhibited impressive performance. However, it encounters two critical issues: 1) The number of prompts grows exponentially as the number of modalities increases; and 2) It lacks robustness in scenarios with different missing modality settings between training and inference. In this paper, we propose a simple yet effective prompt design to address these challenges. Instead of using missing-aware prompts, we utilize prompts as modality-specific tokens, enabling them to capture the unique characteristics of each modality. Furthermore, our prompt design leverages orthogonality between prompts as a key element to learn distinct information across different modalities and promote diversity in the learned representations. Extensive experiments demonstrate that our prompt design enhances both performance and robustness while reducing the number of prompts.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- AV-TAD: Audio-Visual Temporal Action Detection With Transformer2023
- ImageBind One Embedding Space to Bind Them All2023
- Multimodal Prompting with Missing Modalities for Visual Recognition2023
- Visual Prompt Tuning2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Learning to Prompt for Continual Learning2022
- Are Multimodal Transformers Robust to Missing Modality?2022
- Multimodal Transformer with Learnable Frontend and Self Attention for Emotion Recognition2022
- Multimodal Transformer Fusion for Continuous Emotion Recognition2020
- Multimodal human action recognition in assistive human-robot interaction2016