Knowledge-Enhanced Facial Expression Recognition With Emotional-to-Neutral Transformation

Authors:
Publisher website
Google Scholar
Abstract
Existing facial expression recognition (FER) methods typically fine-tune a pre-trained visual encoder using discrete labels. However, this form of supervision limits to specify the emotional concept of different facial expressions. In this paper, we observe that the rich knowledge in text embeddings, generated by vision-language models, is a promising alternative for learning discriminative facial expression representations. Inspired by this, we propose a novel knowledge-enhanced FER method with an emotional-to-neutral transformation. Specifically, we formulate the FER problem as a process to match the similarity between a facial expression representation and text embeddings. Then, we transform the facial expression representation to a neutral representation by simulating the difference in text embeddings from textual facial expression to textual neutral. Finally, a self-contrast objective is introduced to pull the facial expression representation closer to the textual facial expression, while pushing it farther from the neutral representation. We conduct evaluation with diverse pre-trained visual encoders including ResNet-18 and Swin-T on four challenging facial expression datasets. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art FER methods. The code is made publicly available at https://github.com/hangyu94/KE2NT.
Funding Information
  • National Natural Science Foundation of China (U22A2096)
  • National Natural Science Foundation of China (62036007)
  • National Natural Science Foundation of China (62376235)
  • National Natural Science Foundation of China (62306178)
  • Scientific and Technological Innovation Teams in Shaanxi Province (2025RS-CXTD-011)
  • Shaanxi Province Core Technology Research and Development Project (2024QY2-GJHX-11)
  • Fundamental Research Funds for the Central Universities (QTZX23042)
  • GDST Basic Research Fund (2022A1515011652)
  • GDST Basic Research Fund (2024A1515012399)
  • HKBU Faculty Niche Research (RC-FNRA-IG/22-23/SCI/04)
  • HKBU CSD Departmental Incentive Scheme

This publication has 65 references indexed in Scilit: