Jiateng Liu

Micro-expression recognition (MER) is a challenging step in various multimedia applications, such as media understanding and human–computer interaction, as it can reveal genuine human emotions. However, traditional MER often overlooks person-specific facial nuances, limiting generalization and personalized adaptability in practical applications. In this paper, we propose a novel person-specific few-shot MER benchmark that uses only an image/video of the target person as a reference to extend the boundaries of generalization and personalized adaptability in MER. A Prompt-Controlled Multi-Modal Tuning (PCMMT) framework is presented to tackle this benchmark. We introduce a prompt bottleneck mechanism in PCMMT that leverages text prompts and the CLIP multi-modal embedding space to bridge the query and reference visual inputs. Our prompt bottleneck also controls the flow of information across modalities to extract person-specific, subtle micro-expression motions. Moreover, adapter groups are designed at the layer level, with multi-modal tokens fine-tuned separately to refine person-specific cues and enhance generalization. The experimental results show that the proposed person-specific few-shot setting achieves better personalized generalization than traditional MER settings, and our PCMMT outperforms previous state-of-the-art models across various MER benchmarks.

Micro-expression recognition (MER) has a lot of applications in lie detection, education, healthcare, etc., as involuntary micro-expressions (MEs) may provide subtle facial cues associated with affective responses. With the development of deep learning, many studies have recently employed Vision Transformers (ViTs) to investigate MER, since ViTs show promising performance in various visual domains due to their excellent local–global modeling ability. However, such methods confront two fundamental challenges: First, fine-grained visual features are needed to capture the subtle facial movements of MEs, which ViTs relatively fall short on due to coarse patch resolution constrained by their quadratic complexity. Second, the data-intensive nature of ViTs impedes effective learning given the limited scale of ME data. To overcome the aforementioned limitations of using ViTs for MER, we propose the Prior-aware Sparse Transformer (PAST), a novel Transformer-based architecture integrating spatial and semantic prior knowledge synergistically into a sparse attention mechanism, enabling linear-complexity processing of large amounts of fine-grained features. Specifically, we first designed an extraction algorithm to generate a representative set of motion-intensive Principal Anchors, which are used to guide the model’s focus on biologically critical regions during sampling. Second, we introduced the Semantic Dictionary, which was trained with a carefully designed self-contrastive loss to embed task-invariant discriminative semantics of the anchors. Such global semantics further modulate patch sampling and attention weighting in the sparse attention procedure, achieving better training performance with limited ME data. Extensive evaluations on MEGC and CD6ME protocols demonstrate state-of-the-art performance, validating PAST’s efficacy for MER.

Micro-expression recognition (MER) has drawn increasing attention in recent years due to its ability to reveal the true feelings people want to hide. The key challenge in MER is subtle motions, which are hard to capture but crucial for MER. Existing methods usually solve this problem by magnifying all motions in the whole face and temporal sequence. However, micro-expressions (MEs) only involve a few facial areas and several temporal snippets. The all-motion magnification in previous methods cannot precisely capture these local ME motion patterns, and can easily cause spatial as well as temporal distortions, which significantly decrease the MER accuracy. In this paper, we propose an RGB-Frequency Relation Exploration Network (RF-REN), which enhances the subtle motions in refined local ME cues by exploring spatial and temporal relations in both RGB and frequency domains. Specifically, we first decompose the ME video into RGB as well as frequency domains, and conduct temporal division according to different motion stages to cover various ME local patterns. Secondly, we construct an adaptive local-global relation exploration (LGRE) module to explore the local relation cues in the spatial appearance and temporal dynamics in both domains. Finally, we propose an RGB-Frequency routing strategy to fuse the RGB and frequency cues, aiming to aggregate spatial-temporal local-global information and enhance subtle motions for MER. Extensive experiments on three databases (CASME II, SAMM and SMIC) show that the proposed model outperforms other state-of-the-art methods.

A micro-expression (ME) is a kind of involuntary facial expressions, which commonly occurs with subtle intensity. The accurately recognition ME, a. k. a. micro-expression recognition (MER), has a number of potential applications, e.g., interrogation and clinical diagnosis. Therefore, the subject has received a high level of attention among researchers in affective computing and pattern recognition communities. In this paper, we proposed a straightforward and effective deep learning method called uncertainty-aware magnification-robust networks (UAMRN) for MER, which attempts to address two key issues in MER including the low intensity of ME and imbalance of ME samples. Specifically, to better distinguish subtle ME movements, we reconstructed a new sequence by magnifying the ME intensity. Furthermore, a sparse self-attention (SSA) block was implemented which rectifies the standard self-attention with locality sensitive hashing (LSH), resulting in the suppression of artefacts generated during magnification. On the other hand, for the class imbalance problem, we guided the network optimization based on the confidence about the estimation, through which the samples from rare classes were allotted greater uncertainty and thus trained more carefully. We conducted the experiments on three public ME databases, i.e., CASME II, SAMM and SMIC-HS, the results of which demonstrate improvement compared to recent state-of-the-art MER methods.