MCFM: Multimodal Competitive Fusion Mechanism for Sentiment Analysis
- 16 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Social Systems
- Vol. 13 (2) , 1355-1368
Abstract
With the popularity of social media, users are able to express their opinions in multiple forms, such as text, audio, and video. Traditional unimodal sentiment analysis methods can no longer meet the processing requirements of such multisource heterogeneous data, which makes multimodal sentiment analysis a research hotspot. However, most existing methods rely on simple feature splicing or weighted fusion, neglecting the differences in the reliability of different modalities and failing to fully explore the intermodality consistency and difference information. In this article, we propose a multimodal competitive fusion mechanism and construct multimodal competitive fusion model (MCFM). The model first dynamically evaluates the reliability of each modality through the competition mechanism and adaptively assigns weights accordingly. Then it decomposed the modality representations into similar and dissimilar features through modality feature decomposition, supplemented by the overlap of orthogonal traffic channel attention constraints, to achieve the collaborative learning of consistency and dissimilarity features. We evaluate the proposed model on several datasets. In our experiments, we used textual modality data from the dataset with audio modality data for the experiments. The experimental results show that MCFM has 2%–3% higher binary accuracy (ACC2) than the baseline model on the sentiment classification task (with 2% higher binary accuracy under the negative/nonnegative metrics and 3% higher binary accuracy under the positive/negative metrics), and that on the regression task, MCFM’s mean absolute error on the test dataset is 3% lower than that of the baseline model.Keywords
Funding Information
- National Natural Science Foundation of China (62141606)
- National Natural Science Foundation of China (62273101)
- National Natural Science Foundation of China (62236001)
This publication has 55 references indexed in Scilit:
- SSLMM: Semi-Supervised Learning with Missing Modalities for Multimodal Sentiment AnalysisInformation Fusion, 2025
- Cross-modal contrastive learning for multimodal sentiment recognitionApplied Intelligence, 2024
- Progress, achievements, and challenges in multimodal sentiment analysis using deep learning: A surveyApplied Soft Computing, 2023
- TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysisKnowledge-Based Systems, 2023
- Coordinated-joint translation fusion framework with sentiment-interactive graph convolutional networks for multimodal sentiment analysisInformation Processing & Management, 2023
- Disentanglement Translation Network for multimodal sentiment analysisInformation Fusion, 2023
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directionsInformation Fusion, 2022
- RETRACTED ARTICLE: ICDN: integrating consistency and difference networks by transformer for multimodal sentiment analysisApplied Intelligence, 2022
- Multilayer Perceptron (MLP)2017
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008