Transformer-based Multimodal Information Fusion for Facial Expression Analysis
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2427-2436
Abstract
Human affective behavior analysis has received much attention in human-computer interaction (HCI). In this paper, we introduce our submission to the CVPR 2022 Competition on Affective Behavior Analysis in-the-wild (ABAW). To fully exploit affective knowledge from multiple views, we utilize the multimodal features of spoken words, speech prosody, and facial expression, which are extracted from the video clips in the Aff-Wild2 dataset. Based on these features, we propose a unified transformer-based multimodal framework for Action Unit detection and also expression recognition. Specifically, the static vision feature is first encoded from the current frame image. At the same time, we clip its adjacent frames by a sliding window and extract three kinds of multimodal features from the sequence of images, audio, and text. Then, we introduce a transformer-based fusion module that integrates the static vision features and the dynamic multimodal features. The cross-attention module in the fusion module makes the output integrated features focus on the crucial parts that facilitate the downstream detection tasks. We also leverage some data balancing techniques, data augmentation techniques, and postprocessing methods to further improve the model performance. In the official test of ABAW3 Competition, our model ranks first in the EXPR and AU tracks. The extensive quantitative evaluations, as well as ablation studies on the Aff-Wild2 dataset, prove the effectiveness of our proposed method.Keywords
This publication has 29 references indexed in Scilit:
- OpenFace 2.0: Facial Behavior Analysis Toolkit2018
- Recognition of Action Units in the Wild with Deep Nets and a New Global-Local Loss2017
- AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the WildIEEE Transactions on Affective Computing, 2017
- Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge2017
- Learning Spatial and Temporal Cues for Multi-Label Facial Action Unit Detection2017
- Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression RecognitionIEEE Transactions on Affective Computing, 2017
- Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis2016
- Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition2015
- librosa: Audio and Music Signal Analysis in Python2015
- BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression databaseImage and Vision Computing, 2014