TMMDA: A New Token Mixup Multimodal Data Augmentation for Multimodal Sentiment Analysis
Open Access
- 30 April 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1714-1722
Abstract
Existing methods for Multimodal Sentiment Analysis (MSA) mainly focus on integrating multimodal data effectively on limited multimodal data. Learning more informative multimodal representation often relies on large-scale labeled datasets, which are difficult and unrealistic to obtain. To learn informative multimodal representation on limited labeled datasets as more as possible, we proposed TMMDA for MSA, a new Token Mixup Multimodal Data Augmentation, which first generates new virtual modalities from the mixed token-level representation of raw modalities, and then enhances the representation of raw modalities by utilizing the representation of the generated virtual modalities. To preserve semantics during virtual modality generation, we propose a novel cross-modal token mixup strategy based on the generative adversarial network. Extensive experiments on two benchmark datasets, i.e., CMU-MOSI and CMU-MOSEI, verify the superiority of our model compared with several state-of-the-art baselines. The code is available at https://github.com/xiaobaicaihhh/TMMDA.Keywords
This publication has 32 references indexed in Scilit:
- Generative adversarial networksCommunications of the ACM, 2020
- MISA2020
- Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language AnalysisProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Random Erasing Data AugmentationProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Integrating Multimodal Information in Large Pretrained Transformers2020
- Multimodal Transformer for Unaligned Multimodal Language Sequences2019
- Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph2018
- Efficient Low-rank Multimodal Fusion With Modality-Specific Factors2018
- Tensor Fusion Network for Multimodal Sentiment Analysis2017
- Glove: Global Vectors for Word Representation2014