HCHaoyu Chen

Haoyu Chen

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Haoyu Chen has a total of 87 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Point Cloud based Low-Rank Micro-Dynamic Modeling for 4D Micro-Expression Recognition

    by Jinsheng Wei, Jialiang Sun, Wei Mengting, Yante Li, Haoyu Chen, Guanming Lu, Guoying Zhao

  • Open Access

    MAC 2026: Advancing Micro-Action Analysis Towards Fine-Grained Understanding

    by Kun Li, Dan Guo, Jihao Gu, Pengyu Liu, Xiaobai Li, Haoyu Chen, Yanbin Hao, Guoying Zhao, Meng Wang

  • Human Behavior Analysis and Interaction for Emotional Intelligence: Advances, Challenges, and the Road Ahead — EI-MiGA 2026

    by Haoyu Chen, Zhao Ren, Jiadong Wang, Emily Mower Provost, Ehsan Adeli, Björn W. Schuller, Tanja Schultz, Guoying Zhao

  • Evaluating the Adversarial Robustness of Vision–Language Models for Facial Expression Recognition

    by Hui Kuurila-Zhang, Haoyu Chen, Guoying Zhao

    Facial expression recognition (FER) using vision–language models (VLMs) shows strong performance, but their robustness under adversarial conditions is underexplored. Noting that most VLMs rely on CLIP-style vision encoders vulnerable to gradient-based perturbations, we study how attacks on a CLIP encoder affect downstream recognition. We test zero-shot classifiers (CLIP, EVA-CLIP, Exp-CLIP) and generative VLMs (BLIP2, LLaVA, Qwen3-VL) performing FER via visual question answering. Using a CLIP surrogate for gradient-based attacks, we evaluate on AffectNet, RAF-DB, and FERPlus. Key findings include: 1) zero-shot classifiers are highly fragile, 2) attacks transfer only between models sharing the same vision encoder, and 3) generative VLMs are more robust than CLIP variants despite task-agnostic training. These results identify the vision encoder as the main bottleneck and highlight the need for robustness-focused design in future FER systems.


  • Open Access

    SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Prediction?

    by Yueyi Yang, Haotian Liu, Fang Kang, Mengqi Zhang, Zheng Lian, Hao Tang, Haoyu Chen

    We explore the use of large language models (LLMs) for next-utterance prediction in human dialogue. Despite recent advances in LLMs demonstrating their ability to engage in natural conversations with users, we show that even leading models surprisingly struggle to predict a human speaker's next utterance. Instead, humans can readily anticipate forthcoming utterances based on multimodal cues, such as gestures, gaze, and emotional tone, from the context. To systematically examine whether LLMs can reproduce this ability, we propose SayNext-Bench, a benchmark that evaluates LLMs and Multimodal LLMs (MLLMs) on anticipating context-conditioned responses from multimodal cues spanning a variety of real-world scenarios. To support this benchmark, we build SayNext-PC, a novel large-scale dataset containing dialogues with rich multimodal cues. Building on this, we further develop a dual-route prediction MLLM, SayNext-Chat, that incorporates cognitively inspired design to emulate predictive processing in conversation. Experimental results demonstrate that our model outperforms state-of-the-art MLLMs in terms of lexical overlap, semantic similarity, and emotion consistency. Our results prove the feasibility of next-utterance prediction with LLMs from multimodal cues and emphasize the (i) indispensable role of multimodal cues and (ii) actively predictive processing as the foundation of natural human interaction, which is missing in current MLLMs. We hope that this exploration offers a new research entry toward more human-like, context-sensitive AI interaction for human-centered AI. Our benchmark and model can be accessed at https://saynext.github.io/.

  • Open Access

    CROMBO: Cross-Modality Bootstrapping for Unified Sketch–Photo Representation Learning

    by Xingyu Liu, Yan Jiang, Xu Cheng, Hao Yu, Haoyu Chen, Guoying Zhao

    Sketch–photo recognition refers to matching hand-drawn sketches with their corresponding photos, where the performance essentially depends on how well the representations of the two modalities are aligned in the feature spaces. Existing works bluntly force models to reduce the representation discrepancy between the modalities, making the learning less effective. Besides, the current symmetric feature extraction framework prefers the photo modality for richer information while neglecting the sketch modality. Driven by these observations, we argue that, instead of forcefully wiping out the modality discrepancy, we may utilize the discrepancy to enhance model learning. Thus, we propose a Cross-Modality Bootstrapping learning framework (CROMBO) that utilizes the modality discrepancy to bootstrap cross-modality representation learning via a differentiated interaction manner. Specifically, we first present a Sketch Implicit Bootstrapping (SIB) module to magnify the recognizable elements in the photo modality by utilizing the characteristic of sketches having only contours and key details. Second, a Photo-driven Sketch Refinement (PSR) module is developed to guide the sketch representation in the shared feature extraction process by supplementing rich information from the photo modality. Moreover, we design a second-order alignment strategy to dynamically align the latent distribution of two modalities in a Hilbert space. Also, our CROMBO can learn fewer parameters by freezing the weights of shallow layers in the backbone while making no sacrifice in performance. Extensive experiments on six public datasets verify the superior performance of our CROMBO for sketch–photo-based tasks, such as sketch re-identification (Re-ID), sketch–photo face recognition, and sketch-based image retrieval.

  • Open Access

    Detection-Aware Inference for Robust Talking-Head Video Generation

    by Seyyedhamid Azimidokht, Fang Kang, Yueyi Yang, Haoyu Chen


  • Open Access

    LatentMag: Self-Supervised 3D Magnification for Micro Expressions via Latent Extrapolation

    by Mengting Wei, Xingxun Jiang, Haoyu Chen, Yante Li, Guoying Zhao

    Micro-expressions (MEs) are subtle and brief facial movements that reveal genuine emotional states but are often imperceptible due to their low intensity. While motion magnification has proven effective for enhancing ME visibility in 2D settings, its extension to 3D remains largely unexplored. In this work, we present LatentMag, the first controllable 3D micro-expression magnification framework. Unlike traditional editing methods that rely on fixed labels or expression targets, our approach models expression intensity as a relative, input-dependent signal. We adopt registered 3D meshes as our representation, enabling vertex-level correspondence and interpretable displacement analysis. To guide magnification, we introduce a geometric prior that models amplification as a spatially adaptive transformation, where the change in pairwise distance between points on the output mesh scales with that observed between the input shapes, ensuring natural, localized deformation. We operationalize this prior in a generative framework by disentangling a latent intensity code, whose extrapolation drives controllable shape amplification. Trained in a self-supervised manner using unlabeled mesh sequences, LatentMag generalizes well to unseen identities and expressions, offering a novel solution that bridges geometric interpretability with realistic 3D expression modeling.


  • Open Access

    Vision Large Language Models Are Good Noise Handlers in Engagement Analysis

    by Alexander Vedernikov, Puneet Kumar, Haoyu Chen, Tapio Seppänen, Xiaobai Li

    Engagement recognition in video datasets, unlike traditional image classification tasks, is particularly challenged by subjective labels and noise limiting model performance. To overcome the challenges of subjective and noisy engagement labels, we propose a framework leveraging Vision Large Language Models (VLMs) to refine annotations and guide the training process. Our framework uses a questionnaire to extract behavioral cues and split data into high- and low-reliability subsets. We also introduce a training strategy combining curriculum learning with soft label refinement, gradually incorporating ambiguous samples while adjusting supervision to reflect uncertainty. We demonstrate that classical computer vision models trained on refined high-reliability subsets and enhanced with our curriculum strategy show improvements, highlighting benefits of addressing label subjectivity with VLMs. This method surpasses prior state of the art across engagement benchmarks such as EngageNet (three of six feature settings, maximum improvement of +1.21%), and DREAMS / PAFE with F1 gains of +0.22 / +0.06.