DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis
Preprint
Abstract
Multimodal sentiment analysis (MSA) integrates various modalities, such as text, image, and audio, to provide a more comprehensive understanding of sentiment. However, effective MSA is challenged by alignment and fusion issues. Alignment requires synchronizing both temporal and semantic information across modalities, while fusion involves integrating these aligned features into a unified representation. Existing methods often address alignment or fusion in isolation, leading to limitations in performance and efficiency. To tackle these issues, we propose a novel framework called Dual-stream Alignment with Hierarchical Bottleneck Fusion (DashFusion). Firstly, dual-stream alignment module synchronizes multimodal features through temporal and semantic alignment. Temporal alignment employs cross-modal attention to establish frame-level correspondences among multimodal sequences. Semantic alignment ensures consistency across the feature space through contrastive learning. Secondly, supervised contrastive learning leverages label information to refine the modality features. Finally, hierarchical bottleneck fusion progressively integrates multimodal information through compressed bottleneck tokens, which achieves a balance between performance and computational efficiency. We evaluate DashFusion on three datasets: CMU-MOSI, CMU-MOSEI, and CH-SIMS. Experimental results demonstrate that DashFusion achieves state-of-the-art performance across various metrics, and ablation studies confirm the effectiveness of our alignment and fusion techniques. The codes for our experiments are available at https://github.com/ultramarineX/DashFusion.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Triadic temporal-semantic alignment for weakly-supervised video moment retrievalPattern Recognition, 2024
- Efficient Multimodal Transformer With Dual-Level Feature Restoration for Robust Multimodal Sentiment AnalysisIEEE Transactions on Affective Computing, 2023
- Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature FusionIEEE Transactions on Affective Computing, 2023
- A multimodal sentiment analysis system for recognizing person aggressiveness in pain based on textual and visual informationJournal of Ambient Intelligence and Humanized Computing, 2023
- Multimodal sentiment analysis based on fusion methods: A surveyInformation Fusion, 2023
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directionsInformation Fusion, 2022
- Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment AnalysisIEEE Transactions on Affective Computing, 2022
- Multi-source knowledge fusion: a surveyWorld Wide Web, 2020
- Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal MessagesIEEE Intelligent Systems, 2016
- Sentiment Analysis: From Opinion Mining to Human-Agent InteractionIEEE Transactions on Affective Computing, 2015