A Survey on Multimodal Emotion Recognition: Methods, Datasets, and Future Directions
Authors:
- A-Seong MoonAMA-Seong Moon
- Haesung KimHKHaesung Kim
- Ye-Chan ParkYPYe-Chan Park
- Jaesung LeeJaesung Lee
Top Cited Papers
Open Access
- 1 January 2026
- journal article
- Published by Tech Science Press in Computers, Materials & Continua
- Vol. 87 (2) , 1-10
Abstract
Multimodal emotion recognition has emerged as a key research area for enabling human-centered artificial intelligence, supported by the rapid progress in vision, audio, language, and physiological modeling. Existing approaches integrate heterogeneous affective cues through diverse embedding strategies and fusion mechanisms, yet the field remains fragmented due to differences in feature alignment, temporal synchronization, modality reliability, and robustness to noise or missing inputs. This survey provides a comprehensive analysis of MER research from 2021 to 2025, consolidating advances in modality-specific representation learning, cross-modal feature construction, and early, late, and hybrid fusion paradigms. We systematically review visual, acoustic, textual, and sensor-based embeddings, highlighting how pre-trained encoders, self-supervised learning, and large language models have reshaped the representational foundations of MER. We further categorize fusion strategies by interaction depth and architectural design, examining how attention mechanisms, cross-modal transformers, adaptive gating, and multimodal large language models redefine the integration of affective signals. Finally, we summarize major benchmark datasets and evaluation metrics and discuss emerging challenges related to scalability, generalization, and interpretability. This survey aims to provide a unified perspective on multimodal fusion for emotion recognition and to guide future research toward more coherent and generalizable multimodal affective intelligence.Keywords
This publication has 88 references indexed in Scilit:
- Multimodal emotion recognition: A comprehensive review, trends, and challengesWIREs Data Mining and Knowledge Discovery, 2024
- Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamicsMultimedia Tools and Applications, 2024
- Interpretable multimodal emotion recognition using hybrid fusion of speech and image dataMultimedia Tools and Applications, 2023
- Multimodal emotion recognition from facial expression and speech based on feature fusionMultimedia Tools and Applications, 2022
- A novel spatio-temporal convolutional neural framework for multimodal emotion recognitionBiomedical Signal Processing and Control, 2022
- A multimodal emotion recognition model integrating speech, video and MoCAPMultimedia Tools and Applications, 2022
- Speech emotion recognition using data augmentation method by cycle-generative adversarial networksSignal, Image and Video Processing, 2022
- A multimodal emotion recognition method based on facial expressions and electroencephalographyBiomedical Signal Processing and Control, 2021
- Speech Expression Multimodal Emotion Recognition Based on Deep Belief NetworkJournal of Grid Computing, 2021
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008