Attention-Rectified and Texture-Enhanced Cross-Attention Transformer Feature Fusion Network for Facial Expression Recognition
- 6 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 19 (12) , 11823-11832
Abstract
Facial expression recognition (FER) in the wild is a challenging task for affective computing in human–machine interaction fields. However, most of the existing methods fail to learn the most prominent regions of facial images by simple cross-entropy loss due to the imbalance problem commonly existing in FER datasets, which limits the robustness and interpretability of the model. In addition, these methods only capture local features of original images with multisize shallow convolution and ignore facial texture characteristics, leading to a suboptimal recognition performance. To address these issues, in this article, we propose a novel FER network, named the attention-rectified and texture-enhanced cross-attention transformer feature fusion network (AR-TE-CATFFNet). Specifically, an attention-rectified convolution block is first designed to assist multiple convolution heads to focus on the critical areas of human faces and improve the model generalization. Second, we investigate a texture enhancement block to capture texture features through local binary pattern and gray-level co-occurrence matrix, which solves the limitation of insufficient texture information. Finally, a cross-attention transformer feature fusion block is employed to deeply integrate red, green, blue (RGB) features and texture features globally, which is beneficial to boost the accuracy of recognition. Competitive experimental results on three public datasets validate the efficacy of the proposed method, indicating that our proposed method achieves superior classification performance of 89.50% on real-world affective faces database (RAF-DB) dataset, 65.66% on AffectNet dataset, and 74.84% on FER2013 dataset against the existing methods.Keywords
Funding Information
- Beijing Natural Science Foundation (Z220017)
- National Natural Science Foundation of China (62201023)
- Beijing Municipal Education Commission - Natural Science Foundation (KZ202110025036)
- Beijing United Imaging Research Institute of Intelligent Imaging Foundation (CRIBJQY202103)
- Zhejiang Laboratory's International Talent Fund for Young Professionals
This publication has 45 references indexed in Scilit:
- Deep Neural Network Augmentation: Generating Faces for Affect AnalysisInternational Journal of Computer Vision, 2020
- Epileptic Seizure Detection in EEG Signals Using a Unified Temporal-Spectral Squeeze-and-Excitation NetworkIEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020
- Region Attention Networks for Pose and Occlusion Robust Facial Expression RecognitionIEEE Transactions on Image Processing, 2020
- Patch-Gated CNN for Occlusion-aware Facial Expression Recognition2018
- Epileptic Seizure Detection Based on Time-Frequency Images of EEG Signals Using Gaussian Mixture Model and Gray Level Co-Occurrence Matrix FeaturesInternational Journal of Neural Systems, 2018
- AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the WildIEEE Transactions on Affective Computing, 2017
- Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild2017
- EmotioNet: An Accurate, Real-Time Algorithm for the Automatic Annotation of a Million Facial Expressions in the Wild2016
- Glove: Global Vectors for Word Representation2014
- Challenges in Representation Learning: A Report on Three Machine Learning Contests2013