Context-Aware Visual Emotion Recognition Through Hierarchical Fusion of Facial Micro-Features and Scene Semantics
Open Access
- 15 December 2025
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 15 (24) , 13160
Abstract
Visual emotion recognition in unconstrained environments remains challenging, as single-stream deep learning models often fail to capture the localized facial cues and contextual information necessary for accurate classification. This study introduces a hierarchical multi-level feature fusion framework that systematically combines low-level micro-textural features (Local Binary Patterns), mid-level facial cues (Facial Action Units), and high-level scene semantics (Places365) with ResNet-50 global embeddings. Evaluated on the large-scale EmoSet-3.3M dataset, which contains 3.3 million images across eight emotion categories, the framework demonstrates marked performance gains with the best configuration (LBP-FAUs-Places365-ResNet). The proposed framework achieves 74% accuracy and a macro-averaged F1-score of 0.75 under its best configuration (LBP-FAUs-Places365-ResNet), representing a five-percentage-point improvement over the ResNet-50 baseline. The approach excels at distinguishing high-intensity emotions, maintaining efficient inference (2.2 ms per image, 29 M parameters), and analysis confirms that integrating facial muscle activations with scene context enables nuanced emotional differentiation. These results validate that hierarchical feature integration significantly advances robust, human-aligned visual emotion recognition, making it suitable for real-world Human–Computer Interaction (HCI) and affective computing applications.Keywords
This publication has 54 references indexed in Scilit:
- Advancing hierarchical neural networks with scale-aware pyramidal feature learning for medical image dense predictionComputer Methods and Programs in Biomedicine, 2025
- A hybrid feature fusion deep learning framework for multi-source medical image analysisInformation Processing & Management, 2024
- Towards visual emotion analysis via Multi-Perspective Prompt Learning with Residual-Enhanced AdapterKnowledge-Based Systems, 2024
- Multimodal fusion methods with deep neural networks and meta-information for aggression detection in surveillanceExpert Systems with Applications, 2022
- Hierarchical Domain-Adapted Feature Learning for Video Saliency PredictionInternational Journal of Computer Vision, 2021
- Hierarchical dense recursive network for image super-resolutionPattern Recognition, 2020
- Multi-level region-based Convolutional Neural Network for image emotion classificationNeurocomputing, 2019
- Face spoofing detection with local binary pattern networkJournal of Visual Communication and Image Representation, 2018
- Facial Landmark Detection: A Literature SurveyInternational Journal of Computer Vision, 2018
- Computer Vision – ECCV 20142014