A Bayesian Filtering Framework for Continuous Affect Recognition From Facial Images
- 1 April 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 3709-3722
Abstract
Continuous affective state estimation from facial information is a task which requires the prediction of time series of emotional state outputs from a facial image sequence. Modeling the spatial-temporal evolution of facial information plays an important role in affective state estimation. One of the most widely used methods is Recurrent Neural Networks (RNN). RNNs provide an attractive framework for propagating information over a sequence using a continuous-valued hidden layer representation. In this work, we propose to instead learn rich affective state dynamics. We model human affect as a dynamical system and define the affective state in terms of valence, arousal and their higher-order derivatives. We then pose the affective state estimation problem as a jointly trained state estimator for high-dimensional input images, combining an RNN and a Bayesian Filter, i.e. Kalman filters (KF) and Extended Kalman filters (EKF), so that all weights in the resulting network can be trained using backpropagation. We use a recently proposed general framework for designing and learning discriminative state estimators framed as computational graphs. Such approach can handle high dimensional observations and efficiently optimize, in an end-to-end fashion, the state estimator. In addition, to deal with the asynchrony between emotion labels and input images, caused by the inherent reaction lag of the annotators, we introduce a convolutional layer that aligns features with emotion labels. Experimental results, on the RECOLA and SEMAINE datasets for continuous emotion prediction, illustrate the potential of the proposed framework compared to recent state-of-the-art models.Keywords
Funding Information
- China Scholarship Council (201706290115)
- Shaanxi Provincial International Science and Technology Collaboration Project (2017KW-ZD-14)
- VUB Interdisciplinary Research Program
- emo-app project
- Agency for Innovation by Science and Technology in Flanders (131814)
- Vlaamse regering
- Special Construction Fund for Key Disciplines of Shaanxi Provincial Higher Education
This publication has 52 references indexed in Scilit:
- Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological dataPattern Recognition Letters, 2015
- Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical ModalitiesIEEE Transactions on Cybernetics, 2014
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014
- LSTM-Modeling of continuous emotions in an audiovisual affect recognition frameworkImage and Vision Computing, 2013
- AVEC 20122012
- Kalman filters improve LSTM network performance in problems unsolvable by traditional recurrent netsNeural Networks, 2003
- CAPABILITY MEASURES FOR m -DEPENDENT STATIONARY PROCESSESStatistics, 2003
- Long Short-Term MemoryNeural Computation, 1997
- Theory of the Backpropagation Neural Network**Based on “nonindent” by Robert Hecht-Nielsen, which appeared in Proceedings of the International Joint Conference on Neural Networks 1, 593–611, June 1989. © 1989 IEEE.1992
- A circumplex model of affect.Journal of Personality and Social Psychology, 1980