Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
Top Cited Papers
- 14 December 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 28 (5) , 2439-2450
Abstract
Facial expression recognition in the wild is challenging due to various un-constrained conditions. Although existing facial expression classifiers have been almost perfect on analyzing constrained frontal faces, they fail to perform well on partially occluded faces that are common in the wild. In this paper, we propose a Convolution Neutral Network with attention mechanism (ACNN) that can perceive the occlusion regions of the face and focus on the most discriminative unoccluded regions. ACNN is an end to end learning framework. It combines the multiple representations from facial regions of interest (ROIs). Each representation is weighed via a proposed Gate Unit that computes an adaptive weight from the region itself according to the unobstructed-ness and importance. Considering different RoIs, we introduce two versions of ACNN: patch based ACNN (pACNN) and global-local based ACNN (gACNN). pACNN only pays attention to local facial patches. gACNN integrates local representations at patch-level with global representation at image-level. The proposed ACNNs are evaluated on both real and synthetic occlusions, including a self-collected facial expression dataset with real-world occlusions (FED-RO), two largest in-the-wild facial expression datasets (RAF-DB and AffectNet) and their modifications with synthesized facial occlusions. Experimental results show that ACNNs improve the recognition accuracy on both the non-occluded faces and occluded faces. Visualization results demonstrate that, compared with the CNN without Gate Unit, ACNNs are capable of shifting the attention from the occluded patches to other related but unobstructed ones. ACNNs also outperform other state-of-the-art methods on several widely used in-the-lab facial expression datasets under the cross-dataset evaluation protocol.Keywords
Funding Information
- National Key R&D Program of China (2017YFB1002802)
- National Natural Science Foundation of China (61702481, 61702486)
- External Cooperation Program of CAS (GJHZ1843)
This publication has 41 references indexed in Scilit:
- Facial expression recognition using ${l}_{p}$ l p -norm MKL multiclass-SVMMachine Vision and Applications, 2015
- Random Gabor based templates for facial expression recognition in images with facial occlusionNeurocomputing, 2014
- Caffe2014
- Cross-database evaluation for facial expression recognitionPattern Recognition and Image Analysis, 2014
- Facial action unit prediction under partial occlusion based on Error Weighted Cross-Correlation Model2013
- Towards a dynamic expression recognition system under facial occlusionPattern Recognition Letters, 2012
- Facial expression recognition from near-infrared videosImage and Vision Computing, 2011
- Attention control with reinforcement learning for face recognition under partial occlusionMachine Vision and Applications, 2010
- An analysis of facial expression recognition under partial facial image occlusionImage and Vision Computing, 2008
- Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per classIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002