Learning Modality Consistency and Difference Information with Multitask Learning for Multimodal Sentiment Analysis
Open Access
- 17 June 2024
- journal article
- research article
- Published by MDPI AG in Future Internet
- Vol. 16 (6) , 213
Abstract
The primary challenge in Multimodal sentiment analysis (MSA) lies in developing robust joint representations that can effectively learn mutual information from diverse modalities. Previous research in this field tends to rely on feature concatenation to obtain joint representations. However, these approaches fail to fully exploit interactive patterns to ensure consistency and differentiation across different modalities. To address this limitation, we propose a novel framework for multimodal sentiment analysis, named CDML (Consistency and Difference using a Multitask Learning network). Specifically, CDML uses an attention mechanism to assign the attention weights of each modality efficiently. Adversarial training is used to obtain consistent information between modalities. Finally, the difference among the modalities is acquired by the multitask learning framework. Experiments on two benchmark MSA datasets, CMU-MOSI and CMU-MOSEI, showcase that our proposed method outperforms the seven existing approaches by at least 1.3% for Acc-2 and 1.7% for F1.Keywords
Funding Information
- Open Fund of Key Laboratory of Civil Aviation Thermal Hazards Prevention and Emergency Response, Civil Aviation University of China (RZH2021-KF-08, KJZ49420240071)
- Safety Capacity Building Fund Project of the Civil Aviation Administration of China (RZH2021-KF-08, KJZ49420240071)
This publication has 32 references indexed in Scilit:
- A multitask co-training framework for improving speech translation by leveraging speech recognition and machine translation tasksNeural Computing and Applications, 2024
- DJMF: A discriminative joint multi-task framework for multimodal sentiment analysis based on intra- and inter-task dynamicsExpert Systems with Applications, 2023
- A survey on sentiment analysis and its applicationsNeural Computing and Applications, 2023
- CMJRT: Cross-Modal Joint Representation Transformer for Multimodal Sentiment AnalysisIEEE Access, 2022
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directionsInformation Fusion, 2022
- Multimodal Co-learning: Challenges, applications with datasets, recent advances and future directionsInformation Fusion, 2022
- A survey of multimodal sentiment analysisImage and Vision Computing, 2017
- An overview of multi-task learningNational Science Review, 2017
- Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal MessagesIEEE Intelligent Systems, 2016
- Fusing audio, visual and textual clues for sentiment analysis from multimodal contentNeurocomputing, 2016