Adaptive Contextualized Multi-feature Fusion Network for Robust Cross-Linguistic Speech Emotion Recognition
Preprint
- 30 December 2025
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Speech Emotion Recognition (SER) faces significant generalization challenges, particularly in Cross-Linguistic SER (CLSER), due to linguistic and cultural variabilities. Existing approaches struggle with robustly fusing diverse features and adapting to cross-linguistic discrepancies. To address this, we propose the Adaptive Contextualized Multi-feature Fusion Network (ACMF-Net), a novel architecture built on a ``contextualize first, then adaptively fuse'' paradigm. ACMF-Net leverages HuBERT embeddings alongside contextualized Mel-frequency Cepstral Coefficients (MFCCs) and prosodic features, each processed by dedicated Transformer encoders to capture rich temporal dependencies. A core innovation is the Dynamic Gating mechanism, which intelligently learns to dynamically weight the contributions of these heterogeneous feature modalities based on the input speech characteristics, thereby enhancing robustness against cross-linguistic variations. Evaluated on the IEMOCAP dataset for source language performance, ACMF-Net achieved superior Unweighted Accuracy (UAR), outperforming strong baselines and existing multi-feature fusion models. Furthermore, through few-shot fine-tuning on diverse target languages, ACMF-Net consistently demonstrated superior cross-linguistic generalization. An ablation study confirmed the critical contribution of each proposed component, especially the Dynamic Gating mechanism. These results underscore ACMF-Net's potential to significantly advance robust and generalized emotion recognition across linguistic boundaries.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Evaluating Scenario-Based Decision-Making for Interactive Autonomous Driving Using Rational Criteria: A SurveyIEEE Transactions on Intelligent Transportation Systems, 2025
- Uncertainty-Aware Roundabout Navigation: A Switched Decision Framework Integrating Stackelberg Games and Dynamic Potential FieldsIEEE Transactions on Vehicular Technology, 2025
- 3D landmark detection on human point clouds: A benchmark and a dual cascade point transformer frameworkExpert Systems with Applications, 2025
- Visual Gait Alignment for Sensorless Prostheses: Toward an Interpretable Digital Twin FrameworkProceedings of the AAAI Symposium Series, 2025
- LEAF: Unveiling two sides of the same coin in semi-supervised facial expression recognitionComputer Vision and Image Understanding, 2025
- Hybrid data augmentation and deep attention-based dilated convolutional-recurrent neural networks for speech emotion recognitionExpert Systems with Applications, 2023
- Multi-homed abnormal behavior detection algorithm based on fuzzy particle swarm cluster in user and entity behavior analyticsScientific Reports, 2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- CTFN: Hierarchical Learning for Multimodal Sentiment Analysis Using Coupled-Translation Fusion Network2021
- Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection2021