SADGR: Adaptive Cross-Modal Emotion Recognition via Self-Supervised Alignment and Dynamic Gating
Open Access
- 11 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 28229-28244
Abstract
In this paper, we propose SADGR, an adaptive cross-modal emotion recognition framework that follows an “align first, modulate second, and fuse later” paradigm. First, we introduce a self-supervised text–audio contrastive alignment (STA-CA) stage, which leverages large-scale unlabeled audio–text data to map heterogeneous modalities into a shared semantic space prior to fusion. Second, we design an audio-guided visual gating (AG-VR) mechanism, where emotionally salient acoustic cues dynamically modulate visual representations, suppressing redundant information while enhancing emotion-relevant patterns. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that SADGR consistently outperforms state-of-the-art methods across both regression and classification tasks, achieving superior performance in ACC-7, ACC-2, and F1-score while maintaining competitive correlation results. These results indicate that explicit self-supervised alignment combined with dynamic cross-modal gating provides an effective and practical solution for robust multimodal emotion recognition.Keywords
Funding Information
- Henan Provincial Science and Technology Research Project (242102210101)
- Henan Provincial Science and Technology Research Project (252102210214)
- Henan Provincial Key Scientific Research Projects Program for Higher Education Institutions (25A520053)
- Henan Province Intelligent Transportation Video Image Perception and Recognition Engineering Technology Research Center under Grant Yukeshi [2024] No. 1
This publication has 33 references indexed in Scilit:
- TCHFN: Multimodal sentiment analysis based on Text-Centric Hierarchical Fusion NetworkKnowledge-Based Systems, 2024
- TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysisKnowledge-Based Systems, 2023
- VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering2023
- TETFN: A text enhanced transformer fusion network for multimodal sentiment analysisPattern Recognition, 2022
- Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brainPattern Recognition, 2022
- MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering2022
- Wav2CLIP: Learning Robust Audio Representations from Clip2022
- Audioclip: Extending Clip to Image, Text and Audio2022
- Contrastive Representation Learning: A Framework and ReviewIEEE Access, 2020
- A review of affective computing: From unimodal analysis to multimodal fusionInformation Fusion, 2017