Abstract
Open-set Video-based Facial Expression Recognition (OV-FER) extends traditional V-FER to recognize known emotions while detecting unseen ones. A key challenge lies in effectively distinguishing known from unknown categories. Existing CLIP-based approaches focus primarily on positive representations. They often overlook negative semantics, which explicitly models what expressions are not. Negative semantics is crucial for establishing decision boundaries and capturing unknown patterns. However, current negative prompting methods suffer from two limitations. First, negative text prompts fail to encode negation logic due to the standard CLIP text encoder's weakness in handling negation. Second, negative visual prompts lack semantic guidance from positive expression cues, limiting their ability to represent unknown categories. To address these limitations, we propose multimodal NEgative eXpression PROmpting (NEXPRO), a novel framework that fundamentally redesigns negative prompts from both textual and visual perspectives. NEXPRO introduces three synergistic components: (1) Negation Logic Encoding (NLE), which redesigns negative text prompts and fine-tunes the CLIP text encoder to establish clear decision boundaries; (2) Expression-Guided Negative Prompt Learning (EGNPL), which leverages positive expression semantics to guide the optimization of negative visual prompts for better capturing unknown category representations; and (3) Unified Open-set Inference (UOI), which integrates affirmative and contradictory cues into a unified probability framework for robust open-set decision-making. Extensive experiments across four OV-FER tasks on MAFW and AFEW datasets demonstrate that NEXPRO achieves state-of-the-art performance, with significant improvements of +3.39% in AUROC and +2.07% in OSCR on MAFW, and +3.28% and +4.36% on AFEW, respectively, validating its effectiveness.

This publication has 19 references indexed in Scilit: