Abstract
In this paper, we propose SADGR, an adaptive cross-modal emotion recognition framework that follows an “align first, modulate second, and fuse later” paradigm. First, we introduce a self-supervised text–audio contrastive alignment (STA-CA) stage, which leverages large-scale unlabeled audio–text data to map heterogeneous modalities into a shared semantic space prior to fusion. Second, we design an audio-guided visual gating (AG-VR) mechanism, where emotionally salient acoustic cues dynamically modulate visual representations, suppressing redundant information while enhancing emotion-relevant patterns. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that SADGR consistently outperforms state-of-the-art methods across both regression and classification tasks, achieving superior performance in ACC-7, ACC-2, and F1-score while maintaining competitive correlation results. These results indicate that explicit self-supervised alignment combined with dynamic cross-modal gating provides an effective and practical solution for robust multimodal emotion recognition.
Funding Information
  • Henan Provincial Science and Technology Research Project (242102210101)
  • Henan Provincial Science and Technology Research Project (252102210214)
  • Henan Provincial Key Scientific Research Projects Program for Higher Education Institutions (25A520053)
  • Henan Province Intelligent Transportation Video Image Perception and Recognition Engineering Technology Research Center under Grant Yukeshi [2024] No. 1