NEXPRO: Multimodal Negative Expression Prompting for Open-Set Video-based Facial Expression Recognition
Authors:
- Qintai HuQHQintai Hu
- Yifei SuYSYifei Su
- Fan JiangFJFan Jiang
- Xiaodi HuangXiaodi Huang
- Qionghao HuangQHQionghao Huang
- Changqin HuangCHChangqin Huang
Preprint
- 22 April 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Open-set Video-based Facial Expression Recognition (OV-FER) extends traditional V-FER to recognize known emotions while detecting unseen ones. A key challenge lies in effectively distinguishing known from unknown categories. Existing CLIP-based approaches focus primarily on positive representations. They often overlook negative semantics, which explicitly models what expressions are not. Negative semantics is crucial for establishing decision boundaries and capturing unknown patterns. However, current negative prompting methods suffer from two limitations. First, negative text prompts fail to encode negation logic due to the standard CLIP text encoder's weakness in handling negation. Second, negative visual prompts lack semantic guidance from positive expression cues, limiting their ability to represent unknown categories. To address these limitations, we propose multimodal NEgative eXpression PROmpting (NEXPRO), a novel framework that fundamentally redesigns negative prompts from both textual and visual perspectives. NEXPRO introduces three synergistic components: (1) Negation Logic Encoding (NLE), which redesigns negative text prompts and fine-tunes the CLIP text encoder to establish clear decision boundaries; (2) Expression-Guided Negative Prompt Learning (EGNPL), which leverages positive expression semantics to guide the optimization of negative visual prompts for better capturing unknown category representations; and (3) Unified Open-set Inference (UOI), which integrates affirmative and contradictory cues into a unified probability framework for robust open-set decision-making. Extensive experiments across four OV-FER tasks on MAFW and AFEW datasets demonstrate that NEXPRO achieves state-of-the-art performance, with significant improvements of +3.39% in AUROC and +2.07% in OSCR on MAFW, and +3.28% and +4.36% on AFEW, respectively, validating its effectiveness.Keywords
This publication has 19 references indexed in Scilit:
- Prompting multimodal vision-language models for automated student engagement predictionEngineering Applications of Artificial Intelligence, 2026
- Mul-FES: A multimodal facial expression spotting method that integrates textual informationExpert Systems with Applications, 2025
- Learning Sequential Variation Information for Dynamic Facial Expression RecognitionIEEE Transactions on Neural Networks and Learning Systems, 2025
- Unknown Support Prototype Set for Open Set RecognitionInternational Journal of Computer Vision, 2025
- Modeling Fine-Grained Relations in Dynamic Space-Time Graphs for Video-Based Facial Expression RecognitionIEEE Transactions on Affective Computing, 2025
- PTH-Net: Dynamic Facial Expression Recognition Without Face Detection and AlignmentIEEE Transactions on Image Processing, 2024
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Spatial-Temporal Exclusive Capsule Network for Open Set Action RecognitionIEEE Transactions on Multimedia, 2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Learning Open Set Network with Discriminative Reciprocal Points2020