A multiple feature fusion framework for video emotion recognition in the wild
- 7 April 2020
- journal article
- research article
- Published by Wiley in Concurrency and Computation: Practice and Experience
- Vol. 34 (8) , e5764
Abstract
Summary: Human emotions can be recognized from facial expressions captured in videos. It is a growing research area in which many have attempted to improve video emotion detection in both lab‐controlled and unconstrained environments. While existing methods show a decent recognition accuracy on lab‐controlled datasets, they deliver much lower accuracy in a real‐world uncontrolled environment, where a variety of challenges need to be addressed such as variations in illumination, head pose, and individual appearance. Moreover, automatically identifying the key frames consisting of the expression from real‐world videos is another challenge. In this article, to overcome these challenges, we provide a video emotion recognition via multiple feature fusion method. First, a uniform local binary pattern (LBP) and the scale‐invariant feature transform features are extracted from each frame in the video sequences. By applying a random forest classifier, all of the static frames are then labelled by the related emotion class. In this way, the key frames can be automatically identified, including neutral and other expressions. Furthermore, from the key frames, a new geometric feature vector and the LBP from three orthogonal planes are extracted. To further improve robustness, audio features are extracted from the video sequences as an additional dimension to augmenting visual facial expression analysis. The audio and visual features are fused through a kernel multimodal sparse representation. Finally, the corresponding emotion labels to the video sequences can be assigned when a multimodal quality measure specifies the quality of each modality and its role in the decision. The results on both acted facial expressions in the Wild and MMI datasets demonstrate that the proposed method outperforms several counterpart video emotion recognition methods.Keywords
This publication has 50 references indexed in Scilit:
- Automatic face recognition system based on the SIFT featuresComputers and Electrical Engineering, 2015
- Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild2014
- Local Gabor Binary Patterns from Three Orthogonal Planes for Automatic Facial Expression Recognition2013
- Joint Sparse Representation for Robust Multimodal Biometrics RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- A Review of Data Fusion TechniquesThe Scientific World Journal, 2013
- Alternating Direction Algorithms for $\ell_1$-Problems in Compressive SensingSIAM Journal on Scientific Computing, 2011
- An investigation of basic facial expression recognition in autism spectrum disordersCognition and Emotion, 2008
- Developing multimodal intelligent affective interfaces for tele-home health careInternational Journal of Human-Computer Studies, 2003
- Facial Expression Recognition in Adolescents With Mood and Anxiety DisordersAmerican Journal of Psychiatry, 2003
- Generalized Procrustes AnalysisPsychometrika, 1975