Cross-Modal Attention Fusion: A Deep Learning and Affective Computing Model for Emotion Recognition
Open Access
- 24 November 2025
- journal article
- research article
- Published by MDPI AG in Multimodal Technologies and Interaction
- Vol. 9 (12) , 116
Abstract
Artificial emotional intelligence is a sub-domain of human–computer interaction research that aims to develop deep learning models capable of detecting and interpreting human emotional states through various modalities. A major challenge in this domain is identifying meaningful correlations between heterogeneous modalities—for example, between audio and visual data—due to their distinct temporal and spatial properties. Traditional fusion techniques used in multimodal learning to combine data from different sources often fail to adequately capture meaningful and less computational cross-modal interactions, and struggle to adapt to varying modality reliability. Following a review of the relevant literature, this study adopts an experimental research method to develop and evaluate a mathematical cross-modal fusion model, thereby addressing a gap in the extant research literature. The framework uses the Tucker tensor decomposition to analyse the multi-dimensional array of data into a set of matrices to support the integration of temporal features from audio and spatiotemporal features from visual modalities. A cross-attention mechanism is incorporated to enhance cross-modal interaction, enabling each modality to attend to the relevant information from the other. The efficacy of the model is rigorously evaluated on three publicly available datasets and the results conclusively demonstrate that the proposed fusion technique outperforms conventional fusion methods and several more recent approaches. The findings break new ground in this field of study and will be of interest to researchers and developers in artificial emotional intelligence.Keywords
This publication has 74 references indexed in Scilit:
- Audiovisual emotion recognition based on bi-layer LSTM and multi-head attention mechanism on RAVDESS datasetThe Journal of Supercomputing, 2024
- Speech emotion recognition using feature fusion: a hybrid approach to deep learningMultimedia Tools and Applications, 2024
- Multimodal fusion for audio-image and video action recognitionNeural Computing and Applications, 2024
- Interpretable multimodal emotion recognition using hybrid fusion of speech and image dataMultimedia Tools and Applications, 2023
- Deep learning based efficient emotion recognition technique for facial imagesInternational Journal of System Assurance Engineering and Management, 2023
- RETRACTED ARTICLE: Recognition of emotions in speech using deep CNN and RESNETSoft Computing, 2023
- An improved SIFT algorithm for robust emotion recognition under various face poses and illuminationsNeural Computing and Applications, 2019
- Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition2018
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Some Mathematical Notes on Three-Mode Factor AnalysisPsychometrika, 1966