Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition
Authors:
- 17 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4803-4813
Abstract
Though multimodal emotion recognition has achieved significant progress over recent years, the potential of rich synergic relationships across the modalities is not fully exploited. In this paper, we introduce Recursive Joint Cross-Modal Attention (RJCMA) to effectively capture both intra- and inter-modal relationships across audio, visual, and text modalities for dimensional emotion recognition. In particular, we compute the attention weights based on cross-correlation between the joint audio-visual-text feature representations and the feature representations of individual modalities to simultaneously capture intra- and inter-modal relationships across the modalities. The attended features of the individual modalities are again fed as input to the fusion model in a recursive mechanism to obtain more refined feature representations. We have also explored Temporal Convolutional Networks (TCNs) to improve the temporal modeling of the feature representations of individual modalities. Extensive experiments are conducted to evaluate the performance of the proposed fusion model on the challenging Affwild2 dataset. By effectively capturing the synergic intra- and inter-modal relationships across audio, visual, and text modalities, the proposed fusion model achieves a Concordance Correlation Coefficient (CCC) of 0.585 (0.542) and 0.674 (0.619) for valence and arousal, respectively, on the validation set (test set). This shows a significant improvement over the baseline of 0.240 (0.211) and 0.200 (0.191) for valence and arousal, respectively, in the validation set (test set), achieving second place in the valence-arousal challenge of the 6th Affective Behavior Analysis in-the-Wild (ABAW) competition. The code is available on GitHub: https://github.com/praveena2j/RJCMA.Keywords
Funding Information
- Government of Canada
This publication has 56 references indexed in Scilit:
- Audio-Visual Speaker Verification via Joint Cross-Attention2023
- ABAW: Learning from Synthetic Data & Multi-task Learning Challenges2023
- Emotion detection and face recognition of drivers in autonomous vehicles in IoT platformImage and Vision Computing, 2022
- Deep domain adaptation with ordinal regression for pain assessment using weakly-labeled videosImage and Vision Computing, 2021
- Leveraging recent advances in deep learning for audio-Visual emotion recognitionPattern Recognition Letters, 2021
- Transformer models for text-based emotion detection: a review of BERT-based approachesArtificial Intelligence Review, 2021
- Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and BeyondInternational Journal of Computer Vision, 2019
- MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition2016
- LSTM-Modeling of continuous emotions in an audiovisual affect recognition frameworkImage and Vision Computing, 2013
- Three dimensions of emotion.Psychological Review, 1954