HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
Open Access
- 21 January 2025
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 15 (3) , 1053
Abstract
Emotion recognition is becoming increasingly important for accurately understanding and responding to user emotions, driven by the rapid proliferation of non-face-to-face environments and advancements in conversational AI technologies. Existing studies on multimodal emotion recognition, which utilize text and speech, have aimed to improve performance by integrating the information from both modalities. However, these approaches have faced limitations such as restricted information exchange and the omission of critical cues. To address these challenges, this study proposes a Hybrid Multimodal Transformer, which combines Intermediate Layer Fusion and Last Fusion. Text features are extracted using KoELECTRA, while speech features are extracted using HuBERT. These features are processed through a transformer encoder, and Dual Cross Modal Attention is applied to enhance the interaction between text and speech. Finally, the predicted results from each modality are aggregated using an average ensemble method to recognize the final emotion. The experimental results indicate that the proposed model achieves superior emotion recognition performance compared to existing models, demonstrating significant progress in improving both the accuracy and reliability of emotion recognition. In the future, incorporating additional modalities, such as facial expression recognition, is expected to further strengthen multimodal emotion recognition capabilities and open new possibilities for application across diverse fields.Keywords
Funding Information
- Korean government (MSIT) (No. 2023R1A2C1006149)
This publication has 13 references indexed in Scilit:
- Multimodal Speech Emotion Recognition Using Modality-Specific Self-Supervised Frameworks2023
- Emotion Recognition Using Multimodal Fusion Models2023
- Machine Learning Based Integrated Audio and Text Modalities for Enhanced Emotional Analysis2023
- Cross-Modal Fusion Techniques for Utterance-Level Emotion Recognition from Text and Speech2023
- An Empirical Study and Improvement for Speech Emotion Recognition2023
- Multimodal speech emotion recognition based on multi-scale MFCCs and multi-view attention mechanismMultimedia Tools and Applications, 2023
- Changes of teachers’ perception after online distance learning experience due to the COVID-19 pandemicJournal of Educational Technology, 2021
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers2019
- Multimodal Transformer for Unaligned Multimodal Language Sequences2019
- Multimodal Speech Emotion Recognition Using Audio and Text2018