Time-Delay Neural Network for Continuous Emotional Dimension Prediction From Facial Expression Sequences
Open Access
- 21 April 2015
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 46 (4) , 916-929
Abstract
Automatic continuous affective state prediction from naturalistic facial expression is a very challenging research topic but very important in human–computer interaction. One of the main challenges is modeling the dynamics that characterize naturalistic expressions. In this paper, a novel two-stage automatic system is proposed to continuously predict affective dimension values from facial expression videos. In the first stage, traditional regression methods are used to classify each individual video frame, while in the second stage, a time-delay neural network (TDNN) is proposed to model the temporal relationships between consecutive predictions. The two-stage approach separates the emotional state dynamics modeling from an individual emotional state prediction step based on input features. In doing so, the temporal information used by the TDNN is not biased by the high variability between features of consecutive frames and allows the network to more easily exploit the slow changing dynamics between emotional states. The system was fully tested and evaluated on three different facial expression video datasets. Our experimental results demonstrate that the use of a two-stage approach combined with the TDNN to take into account previously classified frames significantly improves the overall performance of continuous emotional state estimation in naturalistic facial expressions. The proposed approach has won the affect recognition sub-challenge of the Third International Audio/Visual Emotion Recognition Challenge 1 . 1It should be noted that only the results of our algorithm were submitted to AVEC2013. Hence, none of the work presented here has been previously published.Keywords
Funding Information
- U.K. Royal Academy of Engineering under Research Exchange with China and India Scheme in 2012
This publication has 79 references indexed in Scilit:
- Fusion of facial expressions and EEG for implicit affective taggingImage and Vision Computing, 2013
- Categorical and dimensional affect analysis in continuous input: Current trends and future directionsImage and Vision Computing, 2013
- Output-associative RVM regression for dimensional and continuous emotion predictionImage and Vision Computing, 2012
- Survey on speech emotion recognition: Features, classification schemes, and databasesPattern Recognition, 2011
- Power PosingPsychological Science, 2010
- The World of Emotions is not Two-DimensionalPsychological Science, 2007
- Multiresolution gray-scale and rotation invariant texture classification with local binary patternsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
- Automatic analysis of facial expressions: the state of the artIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
- Phoneme recognition using time-delay neural networksIEEE Transactions on Acoustics, Speech, and Signal Processing, 1989
- A circumplex model of affect.Journal of Personality and Social Psychology, 1980