Learning Modality-Complementary and Eliminating-Redundancy Representations with Multi-Task Learning for Multimodal Sentiment Analysis
Authors:
- 30 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Abstract
A crucial issue in multimodal language processing is representation learning. Previous works joint training the multimodal and unimodal tasks to learn the consistency and difference of modality representations. However, due to the lack of cross-modal interaction, the extraction of complementary features between modalities is not sufficient. Moreover, during multimodal fusion, the generated multimodal embeddings may be redundant, and unimodal representations also contain noise information, which negatively influence the final sentiment prediction. To this end, we construct a Modality-Complementary and Eliminating-Redundancy multi-task learning model (MCER), and additionally add a cross-modal task to learn complementary features between two modal pairs through gated transformer. Then use two label generation modules to learn modality-specific and modality-complementary representations. Additionally, we introduce the multimodal information bottleneck (MIB) in both multimodal and unimodal tasks to filter out noise information in unimodal representations as well as learn powerful and sufficient multimodal embeddings that is free of redundancy. Last, we conduct extensive experiments on two popular sentiment analysis benchmarks, MOSI and MOSEI. Experimental results demonstrate that our model significantly outperforms the current strong baselines.Keywords
Funding Information
- National Natural Science Foundation of China
This publication has 33 references indexed in Scilit:
- Attention is all you need: utilizing attention in AI-enabled drug discoveryBriefings in Bioinformatics, 2023
- Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis ResearchIEEE Transactions on Affective Computing, 2023
- Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal RepresentationsIEEE Transactions on Multimedia, 2022
- A Survey on Multi-Task LearningIEEE Transactions on Knowledge and Data Engineering, 2021
- Analyzing Multimodal Sentiment Via Acoustic- and Visual-LSTM With Channel-Aware Temporal Convolution NetworkIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Locally Confined Modality Fusion Network With a Global Perspective for Multimodal Human Affective ComputingIEEE Transactions on Multimedia, 2019
- OpenFace 2.0: Facial Behavior Analysis Toolkit2018
- Attribute-Enhanced Face Recognition with Neural Tensor Fusion Networks2017
- Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal MessagesIEEE Intelligent Systems, 2016
- Monte Carlo Sampling Methods2003