Video-based emotion recognition using CNN-RNN and C3D hybrid networks
Authors:
Top Cited Papers
Open Access
- 31 October 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 445-450
Abstract
In this paper, we present a video-based emotion recognition system submitted to the EmotiW 2016 Challenge. The core module of this system is a hybrid network that combines recurrent neural network (RNN) and 3D convolutional networks (C3D) in a late-fusion fashion. RNN and C3D encode appearance and motion information in different ways. Specifically, RNN takes appearance features extracted by convolutional neural network (CNN) over individual video frames as input and encodes motion later, while C3D models appearance and motion of video simultaneously. Combined with an audio module, our system achieved a recognition accuracy of 59.02% without using any additional emotion-labeled video clips in training set, compared to 53.8% of the winner of EmotiW 2015. Extensive experiments show that combining RNN and C3D together can improve video-based emotion recognition noticeably.Keywords
This publication has 24 references indexed in Scilit:
- Long-term recurrent convolutional networks for visual recognition and description2015
- Going deeper with convolutions2015
- Automatic Group Happiness Intensity AnalysisIEEE Transactions on Affective Computing, 2015
- Deep Face Recognition2015
- Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild2014
- Caffe2014
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Combining modality specific deep neural networks for emotion recognition in video2013
- Speech recognition with deep recurrent neural networks2013
- An Application of Recurrent Neural Networks to Discriminative Keyword Spotting2007