CCDA: A Novel Method to Explore the Cross-Correlation in Dual-Attention for Multimodal Sentiment Analysis
Open Access
- 27 February 2024
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 14 (5) , 1934
Abstract
With the development of the Internet, the content that people share contains types of text, images, and videos, and utilizing these multimodal data for sentiment analysis has become an important area of research. Multimodal sentiment analysis aims to understand and perceive emotions or sentiments in different types of data. Currently, the realm of multimodal sentiment analysis faces various challenges, with a major emphasis on addressing two key issues: (1) inefficiency when modeling the intramodality and intermodality dynamics and (2) inability to effectively fuse multimodal features. In this paper, we propose the CCDA (cross-correlation in dual-attention) model, a novel method to explore dynamics between different modalities and fuse multimodal features efficiently. We capture dynamics at intra- and intermodal levels by using two types of attention mechanisms simultaneously. Meanwhile, the cross-correlation loss is introduced to capture the correlation between attention mechanisms. Moreover, the relevant coefficient is proposed to integrate multimodal features effectively. Extensive experiments were conducted on three publicly available datasets, CMU-MOSI, CMU-MOSEI, and CH-SIMS. The experimental results fully confirm the effectiveness of our proposed method, and, compared with the current optimal method (SOTA), our model shows obvious advantages in most of the key metrics, proving its better performance in multimodal sentiment analysis.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61762085, 2019D01C081)
- Natural Science Foundation of Xinjiang Uygur Autonomous Region Project (61762085, 2019D01C081)
This publication has 43 references indexed in Scilit:
- Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature FusionIEEE Transactions on Affective Computing, 2023
- Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis ResearchIEEE Transactions on Affective Computing, 2023
- TEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment AnalysisCognitive Computation, 2022
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directionsInformation Fusion, 2022
- Learning to Learn Better Unimodal Representations via Adaptive Multimodal Meta-LearningIEEE Transactions on Affective Computing, 2022
- The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and ImprovementsIEEE Transactions on Affective Computing, 2021
- Computational Media Intelligence: Human-Centered Machine Analysis of MediaProceedings of the IEEE, 2021
- A survey of multimodal sentiment analysisImage and Vision Computing, 2017
- A review of affective computing: From unimodal analysis to multimodal fusionInformation Fusion, 2017
- Multimodal fusion for multimedia analysis: a surveyMultimedia Systems, 2010