Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models
Preprint
Abstract
Multimodal learning typically relies on the assumption that all modalities are fully available during both the training and inference phases. However, in real-world scenarios, consistently acquiring complete multimodal data presents significant challenges due to various factors. This often leads to the issue of missing modalities, where data for certain modalities are absent, posing considerable obstacles not only for the availability of multimodal pretrained models but also for their fine-tuning and the preservation of robustness in downstream tasks. To address these challenges, we propose a novel framework integrating parameter-efficient fine-tuning of unimodal pretrained models with a self-supervised joint-embedding learning method. This framework enables the model to predict the embedding of a missing modality in the representation space during inference. Our method effectively predicts the missing embedding through prompt tuning, leveraging information from available modalities. We evaluate our approach on several multimodal benchmark datasets and demonstrate its effectiveness and robustness across various scenarios of missing modalities.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2022
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- SimCSE: Simple Contrastive Learning of Sentence Embeddings2021
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021
- UNITER: UNiversal Image-TExt Representation Learning2020
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers2019