Happy Emotion Recognition From Unconstrained Videos Using 3D Hybrid Deep Features
Open Access
- 24 February 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 35524-35538
Abstract
Facial expressions have been proven to be the most effective way for the brain to recognize human emotions in a variety of contexts. With the exponentially increasing research for emotion detection in recent years, facial expression recognition has become an attractive, hot research topic to identify various basic emotions. Happy emotion is one of such basic emotions with many applications, which is more likely recognized by facial expressions than other emotion measurement instruments (e.g., audio/speech, textual and physiological sensing). Nowadays, most methods have been developed for identifying multiple types of emotions, which aim to achieve the best overall precision for all emotions; it is hard for them to optimize the recognition accuracy for single emotion (e.g., happiness). Only a few methods are designed to recognize single happy emotion captured in the unconstrained videos; however, their limitations lie in that the processing of severe head pose variations has not been considered, and the accuracy is still not satisfied. In this paper, we propose a Happy Emotion Recognition model using the 3D hybrid deep and distance features (HappyER-DDF) method to improve the accuracy by utilizing and extracting two different types of deep visual features. First, we employ a hybrid 3D Inception-ResNet neural network and long-short term memory (LSTM) to extract dynamic spatial-temporal features among sequential frames. Second, we detect facial landmarks' features and calculate the distance between each facial landmark and a reference point on the face (e.g., nose peak) to capture their changes when a person starts to smile (or laugh). We implement the experiments using both feature-level and decision-level fusion techniques on three unconstrained video datasets. The results demonstrate that our HappyER-DDF method is arguably more accurate than several currently available facial expression models.Keywords
Funding Information
- Australia Research Council (ARC) Discovery Project (DP190100587)
This publication has 57 references indexed in Scilit:
- Facial expression recognition in the wild based on multimodal texture featuresJournal of Electronic Imaging, 2016
- Collaborative filtering with facial expressions for online video recommendationInternational Journal of Information Management, 2016
- Three-dimensional (3D) facial recognition and predictionSignal, Image and Video Processing, 2016
- A new descriptor of gradients Self-Similarity for smile detection in unconstrained scenariosNeurocomputing, 2016
- Local Gabor Binary Patterns from Three Orthogonal Planes for Automatic Facial Expression Recognition2013
- Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected "In-the-Wild"2013
- A Review of Data Fusion TechniquesThe Scientific World Journal, 2013
- Smile detection by boosting pixel differencesIEEE Transactions on Image Processing, 2011
- A preference for genuine smiles following social exclusionJournal of Experimental Social Psychology, 2010
- Toward Practical Smile DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2009