Jiabei Zeng
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jiabei Zeng has a total of 41 co-authors. Here are the top co-authors by number of shared publications:
- Shiguang Shan
Institute of Computing Technology, Chinese Academy of Sciences
11 shared publications Zhang Xiong
Engineering Research Center of Advanced Computer Application Technology, Ministry of Education, Beihang University, Beijing, China
5 shared publications- Biao Leng
Beihang University
4 shared publications Shiguang Shan
Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
4 shared publications- Xilin Chen
Institute of Computing Technology, Chinese Academy of Sciences
4 shared publications - Fei Chang
Institute of Computing Technology, Chinese Academy of Sciences, China and Peng Cheng Laboratory, China
3 shared publications
Latest Publications
- Open Access
- 2 August 2026
- Published by Elsevier in Pattern Recognition
- Vol. 180
CoSI-Gaze: Context-Spatial Integration for gaze target detection and social gaze prediction
- 2 August 2026
- Published by Elsevier in Pattern Recognition
- Vol. 180
Open AccessCoSI-Gaze: Context-Spatial Integration for gaze target detection and social gaze prediction
- 6 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 33 (10709908) , 938-942
Facial Action Units Generation via Cross-Modality Attention Fusion and Calibrated Denoising
The advances in diffusion models have enabled high-quality facial image generation. Facial Action Units (AUs), which systematically represent fine-grained expressions through individual facial muscle activations, have recently been incorporated to enable fine-grained face generation. However, existing AU-conditioned methods still face notable limitations: (1) interference between AU and text prompt during condition injection, and (2) absence of AU-specific supervision. To ensure precise expression control and prompt alignment, we propose two complementary modules in the training and inference phases, respectively. The Cross-modality Attention Fusion (CAF) module integrates AU and text features at the attention level during training, mitigating interference and jointly optimizing both expression control and semantic alignment objectives. The Detector-Guided Optimization (DGO) module corrects AU deviations during inference by using a pretrained AU detector to guide each denoising step. This training-free module guides the generation toward the desired AU intensities, improving expression accuracy without affecting identity. Extensive experiments show that our method achieves superior AU alignment and semantic consistency, outperforming prior works on AU MSE, CLIP-I, and FID in all settings.
- 6 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 33 (10709908) , 938-942
Facial Action Units Generation via Cross-Modality Attention Fusion and Calibrated Denoising
The advances in diffusion models have enabled high-quality facial image generation. Facial Action Units (AUs), which systematically represent fine-grained expressions through individual facial muscle activations, have recently been incorporated to enable fine-grained face generation. However, existing AU-conditioned methods still face notable limitations: (1) interference between AU and text prompt during condition injection, and (2) absence of AU-specific supervision. To ensure precise expression control and prompt alignment, we propose two complementary modules in the training and inference phases, respectively. The Cross-modality Attention Fusion (CAF) module integrates AU and text features at the attention level during training, mitigating interference and jointly optimizing both expression control and semantic alignment objectives. The Detector-Guided Optimization (DGO) module corrects AU deviations during inference by using a pretrained AU detector to guide each denoising step. This training-free module guides the generation toward the desired AU intensities, improving expression accuracy without affecting identity. Extensive experiments show that our method achieves superior AU alignment and semantic consistency, outperforming prior works on AU MSE, CLIP-I, and FID in all settings.
- 17 October 2025
- Published by Springer Nature in Frontiers of Computer Science
- Vol. 20 (2) , 1-3
Task-aware pre-training for facial action analysis
- 17 October 2025
- Published by Springer Nature in Frontiers of Computer Science
- Vol. 20 (2) , 1-3
Task-aware pre-training for facial action analysis
- 19 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (1) , 62-78
Unsupervised Gaze Representation Learning by Switching Features
It is prevalent to leverage unlabeled data to train deep learning models when it is difficult to collect large-scale annotated datasets. However, for 3D gaze estimation, most existing unsupervised learning methods face challenges in distinguishing subtle gaze-relevant information from dominant gaze-irrelevant information. To address this issue, we propose an unsupervised learning framework to disentangle the gaze-relevant and the gaze-irrelevant information, by seeking the shared information of a pair of input images with the same gaze and with the same eye respectively. Specifically, given two images, the framework finds their shared information by first encoding the images into two latent features via two encoders and then switching part of the features before feeding them to the decoders for image reconstruction. We theoretically prove that the proposed framework is able to encode different information into different parts of the latent feature if we properly select the training image pairs and their shared information. Based on the framework, we derive Cross-Encoder and Cross-Encoder++ to learn gaze representation from the eye images and face images, respectively. Experiments on public gaze datasets demonstrate that the Cross-Encoder and Cross-Encoder++ outperform the competitive methods. The ablation study quantitatively and qualitatively shows that the gaze feature is successfully extracted.
- 19 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (1) , 62-78
Unsupervised Gaze Representation Learning by Switching Features
It is prevalent to leverage unlabeled data to train deep learning models when it is difficult to collect large-scale annotated datasets. However, for 3D gaze estimation, most existing unsupervised learning methods face challenges in distinguishing subtle gaze-relevant information from dominant gaze-irrelevant information. To address this issue, we propose an unsupervised learning framework to disentangle the gaze-relevant and the gaze-irrelevant information, by seeking the shared information of a pair of input images with the same gaze and with the same eye respectively. Specifically, given two images, the framework finds their shared information by first encoding the images into two latent features via two encoders and then switching part of the features before feeding them to the decoders for image reconstruction. We theoretically prove that the proposed framework is able to encode different information into different parts of the latent feature if we properly select the training image pairs and their shared information. Based on the framework, we derive Cross-Encoder and Cross-Encoder++ to learn gaze representation from the eye images and face images, respectively. Experiments on public gaze datasets demonstrate that the Cross-Encoder and Cross-Encoder++ outperform the competitive methods. The ablation study quantitatively and qualitatively shows that the gaze feature is successfully extracted.
- Open Access
Pose-Robust Calibration Strategy for Point-of-Gaze Estimation on Mobile Phones
Although appearance-based point-of-gaze (PoG) estimation has improved, the estimators still struggle to generalize across individuals due to personal differences. Therefore, person-specific calibration is required for accurate PoG estimation. However, calibrated PoG estimators are often sensitive to head pose variations. To address this, we investigate the key factors influencing calibrated estimators and explore pose-robust calibration strategies. Specifically, we first construct a benchmark, MobilePoG, which includes facial images from 32 individuals focusing on designated points under either fixed or continuously changing head poses. Using this benchmark, we systematically analyze how the diversity of calibration points and head poses influences estimation accuracy. Our experiments show that introducing a wider range of head poses during calibration improves the estimator's ability to handle pose variation. Building on this insight, we propose a dynamic calibration strategy in which users fixate on calibration points while moving their phones. This strategy naturally introduces head pose variation during a user-friendly and efficient calibration process, ultimately producing a better calibrated PoG estimator that is less sensitive to head pose variations than those using conventional calibration strategies. Codes and datasets are available at our project page.
Open AccessPose-Robust Calibration Strategy for Point-of-Gaze Estimation on Mobile Phones
Although appearance-based point-of-gaze (PoG) estimation has improved, the estimators still struggle to generalize across individuals due to personal differences. Therefore, person-specific calibration is required for accurate PoG estimation. However, calibrated PoG estimators are often sensitive to head pose variations. To address this, we investigate the key factors influencing calibrated estimators and explore pose-robust calibration strategies. Specifically, we first construct a benchmark, MobilePoG, which includes facial images from 32 individuals focusing on designated points under either fixed or continuously changing head poses. Using this benchmark, we systematically analyze how the diversity of calibration points and head poses influences estimation accuracy. Our experiments show that introducing a wider range of head poses during calibration improves the estimator's ability to handle pose variation. Building on this insight, we propose a dynamic calibration strategy in which users fixate on calibration points while moving their phones. This strategy naturally introduces head pose variation during a user-friendly and efficient calibration process, ultimately producing a better calibrated PoG estimator that is less sensitive to head pose variations than those using conventional calibration strategies. Codes and datasets are available at our project page.
- 14 May 2025
- Published by Elsevier in Pattern Recognition
- Vol. 168
Exp-VQA: Fine-grained facial expression analysis via visual question answering
- 14 May 2025
- Published by Elsevier in Pattern Recognition
- Vol. 168
Exp-VQA: Fine-grained facial expression analysis via visual question answering
- 18 September 2023
- Published by Elsevier in Pattern Recognition
- Vol. 146
Gaze estimation with semi-supervised eye landmark detection as an auxiliary task
- 18 September 2023
- Published by Elsevier in Pattern Recognition
- Vol. 146
Gaze estimation with semi-supervised eye landmark detection as an auxiliary task
- Open Access
- 30 May 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Gaze Pattern Recognition in Dyadic Communication
Analyzing gaze behaviors is crucial to interpret the nature of communication. Current studies on gaze have focused primarily on the detection of a single pattern, such as the Looking-At-Each-Other pattern or the shared attention pattern. In this work, we re-define five static gaze patterns that cover all the status during a dyadic communication and propose a network to recognize these mutual exclusive gaze patterns given an image. We annotate a benchmark, called GP-Static, for the gaze pattern recognition task, on which our method experimentally outperforms other alternate solutions. Our method also achieves the state-of-art performance on other two single gaze pattern recognition tasks. The analysis of gaze patterns on preschool children demonstrates that the statistic of the proposed static gaze patterns conforms with the findings in psychology.
- 30 May 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessGaze Pattern Recognition in Dyadic Communication
Analyzing gaze behaviors is crucial to interpret the nature of communication. Current studies on gaze have focused primarily on the detection of a single pattern, such as the Looking-At-Each-Other pattern or the shared attention pattern. In this work, we re-define five static gaze patterns that cover all the status during a dyadic communication and propose a network to recognize these mutual exclusive gaze patterns given an image. We annotate a benchmark, called GP-Static, for the gaze pattern recognition task, on which our method experimentally outperforms other alternate solutions. Our method also achieves the state-of-art performance on other two single gaze pattern recognition tasks. The analysis of gaze patterns on preschool children demonstrates that the statistic of the proposed static gaze patterns conforms with the findings in psychology.
- 21 July 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 44 (1) , 302-317
Learning Representations for Facial Actions From Unlabeled Videos
Facial actions are usually encoded as anatomy-based action units (AUs), the labelling of which demands expertise and thus is time-consuming and expensive. To alleviate the labelling demand, we propose to leverage the large number of unlabelled videos by proposing a Twin-cycle Autoencoder (TAE) to learn discriminative representations for facial actions. TAE is inspired by the fact that facial actions are embedded in the pixel-wise displacements between two sequential face images (hereinafter, source and target) in the video. Therefore, learning the representations of facial actions can be achieved by learning the representations of the displacements. However, the displacements induced by facial actions are entangled with those induced by head motions. TAE is thus trained to disentangle the two kinds of movements by evaluating the quality of the synthesized images when either the facial actions or head pose is changed, aiming to reconstruct the target image. Experiments on AU detection show that TAE can achieve accuracy comparable to other existing AU detection methods including some supervised methods, thus validating the discriminant capacity of the representations learned by TAE. TAE's ability in decoupling the action-induced and pose-induced movements is also validated by visualizing the generated images and analyzing the facial image retrieval results qualitatively and quantitatively.
- 21 July 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 44 (1) , 302-317
Learning Representations for Facial Actions From Unlabeled Videos
Facial actions are usually encoded as anatomy-based action units (AUs), the labelling of which demands expertise and thus is time-consuming and expensive. To alleviate the labelling demand, we propose to leverage the large number of unlabelled videos by proposing a Twin-cycle Autoencoder (TAE) to learn discriminative representations for facial actions. TAE is inspired by the fact that facial actions are embedded in the pixel-wise displacements between two sequential face images (hereinafter, source and target) in the video. Therefore, learning the representations of facial actions can be achieved by learning the representations of the displacements. However, the displacements induced by facial actions are entangled with those induced by head motions. TAE is thus trained to disentangle the two kinds of movements by evaluating the quality of the synthesized images when either the facial actions or head pose is changed, aiming to reconstruct the target image. Experiments on AU detection show that TAE can achieve accuracy comparable to other existing AU detection methods including some supervised methods, thus validating the discriminant capacity of the representations learned by TAE. TAE's ability in decoupling the action-induced and pose-induced movements is also validated by visualizing the generated images and analyzing the facial image retrieval results qualitatively and quantitatively.