Deep Learning for Spatio-Temporal Modeling of Dynamic Spontaneous Emotions
- 4 October 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 12 (2) , 363-376
Abstract
Facial expressions involve dynamic morphological changes in a face, conveying information about the expresser's feelings. Each emotion has a specific spatial deformation over the face and temporal profile with distinct time segments. We aim at modeling the human dynamic emotional behavior by taking into consideration the visual content of the face and its evolution. But emotions can both speed-up or slow-down, therefore it is important to incorporate information from the local neighborhood frames (short-term dependencies) and the global setting (long-term dependencies) to summarize the segment context despite of its time variations. A 3D-Convolutional Neural Networks (3D-CNN) is used to learn early local spatiotemporal features. The 3D-CNN is designed to capture subtle spatiotemporal changes that may occur on the face. Then a Convolutional-Long-Short-Term-Memory (ConvLSTM) network is designed to learn semantic information by taking into account longer spatiotemporal dependencies. The ConvLSTM network helps considering the global visual saliency of the expression. That is locating and learning features in space and time that stand out from their local neighbors in order to signify distinctive facial expression features along the entire sequence. Non-variant representations based on aggregating global spatiotemporal features at increasingly fine resolutions are then done using a weighted Spatial Pyramid Pooling layer.Keywords
This publication has 37 references indexed in Scilit:
- Dynemo : A Video Database of Natural Facial Expressions of EmotionsThe International journal of Multimedia & Its Applications, 2013
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Facial Expression Recognition Using Spatiotemporal Boosted Discriminatory Classifiers2010
- Machine analysis of facial behaviour: naturalistic and dynamic behaviourPhilosophical Transactions of the Royal Society B: Biological Sciences, 2009
- Facial expression recognition based on Local Binary Patterns: A comprehensive studyImage and Vision Computing, 2009
- Dynamics of facial expression extracted automatically from videoImage and Vision Computing, 2006
- Multidimensional Assessment of Emotion Regulation and Dysregulation: Development, Factor Structure, and Initial Validation of the Difficulties in Emotion Regulation ScaleJournal of Psychopathology and Behavioral Assessment, 2004
- Long Short-Term MemoryNeural Computation, 1997
- Facial expression and emotion.American Psychologist, 1993
- Turing computability with neural netsApplied Mathematics Letters, 1991