Covariance Pooling for Facial Expression Recognition
Authors:
- Dinesh AcharyaDADinesh Acharya
- Zhiwu HuangZHZhiwu Huang
- Danda Pani PaudelDPDanda Pani Paudel
- Luc Van GoolLGLuc Van Gool
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 480-4807
Abstract
Classifying facial expressions into different categories requires capturing regional distortions of facial landmarks. We believe that second-order statistics such as covariance is better able to capture such distortions in regional facial features. In this work, we explore the benefits of using a manifold network structure for covariance pooling to improve facial expression recognition. In particular, we first employ such kind of manifold networks in conjunction with traditional convolutional networks for spatial pooling within individual image feature maps in an end-to-end deep learning manner. By doing so, we are able to achieve a recognition accuracy of 58.14% on the validation set of Static Facial Expressions in the Wild (SFEW2.0) and 87.0% on the validation set of Real-World Affective Faces (RAF) Database1. Both of these results are the best results we are aware of. Besides, we leverage covariance pooling to capture the temporal evolution of per-frame features for video-based facial expression recognition. Our reported results demonstrate the advantage of pooling image-set features temporally by stacking the designed manifold network of covariance pooling on top of convolutional network layers.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Video-based emotion recognition using CNN-RNN and C3D hybrid networks2016
- Emotion recognition in the wild from videos using images2016
- Multi-clue fusion for emotion recognition in the wild2016
- Joint Face Detection and Alignment Using Multitask Cascaded Convolutional NetworksIEEE Signal Processing Letters, 2016
- EmotioNet: An Accurate, Real-Time Algorithm for the Automatic Annotation of a Million Facial Expressions in the Wild2016
- Matrix Backpropagation for Deep Networks with Structured Layers2015
- Challenges in representation learning: A report on three machine learning contestsNeural Networks, 2015
- Emotion Recognition In The Wild Challenge 20142014
- Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild2014
- Comprehensive database for facial expression analysis2002