Automatic emotion recognition in the wild using an ensemble of static and dynamic representations
Authors:
Open Access
- 31 October 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 514-521
Abstract
Automatic emotion recognition in the wild video datasets is a very challenging problem because of the inter-class similarities among different facial expressions and large intra-class variabilities due to the significant changes in illumination, pose, scene, and expression. In this paper, we present our proposed method for video-based emotion recognition in the EmotiW 2016 challenge. The task considers the unconstrained emotion recognition problem by training from short video clips extracted from movies and testing on short movie clips and spontaneous video clips of the reality TV data. Four different methods are employed to extract both static and dynamic emotion representations from the videos. First, local binary patterns of three orthogonal planes are used to describe spatiotemporal features of the video frames. Second, principal component analysis is applied to the image patches in a two-stage convolutional network to learn weights and extract facial features from the aligned faces. Third, the deep convolutional neural network model of VGG-Face is deployed to extract deep facial representations from aligned faces. Fourth, a bag of visual words is computed based on dense scale-invariant feature transform descriptors from aligned face images to form hand-crafted representations. Support vector machines are then utilized to train and classify the obtained spatiotemporal representations and facial features. Finally, score-level fusion is applied to combine the classification results and predict the emotion labels of the video clips. The results show that the proposed combined method has outperformed all the utilized techniques with the overall validations and test accuracies of 43.13% and 40.13%, respectively. This system, is relatively a good classifier in Happy and Angry emotion categories and is unsuccessful in detecting Surprise, Disgust, and Fear.Keywords
Funding Information
- TÜBİTAK (113E067)
- Marie Curie 7th EU Framework Programme (MMFP)
This publication has 24 references indexed in Scilit:
- AV+EC 20152015
- PCANet: A Simple Deep Learning Baseline for Image Classification?IEEE Transactions on Image Processing, 2015
- Emotion Recognition In The Wild Challenge 20142014
- Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild2014
- Why is facial expression analysis in the wild challenging?2013
- Combining modality specific deep neural networks for emotion recognition in video2013
- Supervised Descent Method and Its Applications to Face Alignment2013
- Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories2006
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- Comprehensive database for facial expression analysis2002