Ruijie Quan
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Ruijie Quan has a total of 27 co-authors. Here are the top co-authors by number of shared publications:
- Yi Yang
Zhejiang University, China
15 shared publications - Wenguan Wang
Zhejiang University
6 shared publications - Linchao Zhu
College of Computer Science and Technology Zhejiang University
5 shared publications - Tuo Feng
Zhejiang University
3 shared publications - Chao Wang
CSIRO, Sydney, NSW, Australia and University of Technology Sydney, Sydney, NSW, Australia
2 shared publications - Hehe Fan
Zhejiang University, Hangzhou, China
2 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by MDPI AG in Preprints.org
A Survey of AI Virtual Cells: Representation Learning, Perturbation Prediction, and Intervention Design
Predicting cellular responses to genetic and chemical interventions supports disease research and therapeutic discovery. Experimental screening is costly, limiting coverage of interventions and cellular contexts. This survey reviews artificial intelligence virtual cells (AIVC) through the progression from state representation learning to perturbation prediction and intervention design. We describe how state representations support response modeling and how learned responses guide intervention selection across previously untested interventions and diverse cellular contexts. We summarize data resources, representative model families, evaluation strategies, and multimodal, temporal, and spatial extensions, then discuss generalization and experimental feedback. A curated resource collection is available at https://github.com/ttruan2426-dot/Awesome-Single-Cell-Perturbation-Prediction.
- 9 November 2026
- Published by MDPI AG in Preprints.org
Open AccessA Survey of AI Virtual Cells: Representation Learning, Perturbation Prediction, and Intervention Design
Predicting cellular responses to genetic and chemical interventions supports disease research and therapeutic discovery. Experimental screening is costly, limiting coverage of interventions and cellular contexts. This survey reviews artificial intelligence virtual cells (AIVC) through the progression from state representation learning to perturbation prediction and intervention design. We describe how state representations support response modeling and how learned responses guide intervention selection across previously untested interventions and diverse cellular contexts. We summarize data resources, representative model families, evaluation strategies, and multimodal, temporal, and spatial extensions, then discuss generalization and experimental feedback. A curated resource collection is available at https://github.com/ttruan2426-dot/Awesome-Single-Cell-Perturbation-Prediction.
- 14 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-15
Mask-Guided Self-Supervised Video Object Segmentation
This work aims to learn video object segmentation (mask propagation) in a self-supervised manner. We develop a unified framework which simultaneously models cross-frame dense correspondence for locally discriminative feature learning and embeds object-level context for target-mask decoding. As a result, it is able to directly learn to perform mask-guided sequential segmentation from unlabeled videos, in contrast to previous efforts usually relying on an oblique solution — cheaply “copying” labels according to pixel-wise correlations. Concretely, our algorithm alternates between i) clustering video pixels for creating pseudo segmentation labels ex nihilo; and ii) utilizing the pseudo labels to learn mask encoding and decoding for VOS. Unsupervised correspondence learning is further incorporated into this self-taught, mask embedding scheme, so as to ensure the generic nature of the learnt representation and avoid cluster degeneracy. Building on the above design, we further lift the offline clustering process to an online version, which streamlines the integration of mask embedding and dense correspondence modeling, thus resulting in a more cohesive online learning framework. Consequently, this facilitates more effective generation of pseudo labels without compromising the training speed. Additionally, we introduce a semantic centroids pool, a repository of content-aware visual representations across the entire video data. Managed through multi-frame clustering centroids, it is able to increase the precision and reliability of pseudo segmentation labels. Our improved algorithm sets state-of-the-arts on three standard benchmarks (ie, DAVIS$_{17}$, YouTube-VOS, and VIP), narrowing the gap between self- and fully-supervised VOS. Code is available at Mask-VOS.
- 14 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-15
Mask-Guided Self-Supervised Video Object Segmentation
This work aims to learn video object segmentation (mask propagation) in a self-supervised manner. We develop a unified framework which simultaneously models cross-frame dense correspondence for locally discriminative feature learning and embeds object-level context for target-mask decoding. As a result, it is able to directly learn to perform mask-guided sequential segmentation from unlabeled videos, in contrast to previous efforts usually relying on an oblique solution — cheaply “copying” labels according to pixel-wise correlations. Concretely, our algorithm alternates between i) clustering video pixels for creating pseudo segmentation labels ex nihilo; and ii) utilizing the pseudo labels to learn mask encoding and decoding for VOS. Unsupervised correspondence learning is further incorporated into this self-taught, mask embedding scheme, so as to ensure the generic nature of the learnt representation and avoid cluster degeneracy. Building on the above design, we further lift the offline clustering process to an online version, which streamlines the integration of mask embedding and dense correspondence modeling, thus resulting in a more cohesive online learning framework. Consequently, this facilitates more effective generation of pseudo labels without compromising the training speed. Additionally, we introduce a semantic centroids pool, a repository of content-aware visual representations across the entire video data. Managed through multi-frame clustering centroids, it is able to increase the precision and reliability of pseudo segmentation labels. Our improved algorithm sets state-of-the-arts on three standard benchmarks (ie, DAVIS$_{17}$, YouTube-VOS, and VIP), narrowing the gap between self- and fully-supervised VOS. Code is available at Mask-VOS.
- 24 December 2025
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 134 (1) , 12
Audio-Guided Video Scene Editing
- 24 December 2025
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 134 (1) , 12
Audio-Guided Video Scene Editing
- Open Access
- 7 November 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 21 (11) , 1-21
ExpAvatar: High-Fidelity Avatar Generation of Unseen Expressions with 3D Face Priors
The reconstruction of dynamic head avatars has gained increasing significance, giving rise to various downstream applications such as visual dubbing and digital human creation. Despite recent advancements, generating novel, unseen expressions for a given identity remains challenging in concurrently achieving (1) accurate expression and consistent appearance and (2) high-quality and realistic faces. This article introduces ExpAvatar, a novel approach crafted to address these challenges. ExpAvatar elaborately leverages the appearance consistency capabilities inherent in 3DMMs-based models along with the robust generalization ability of DDPMs-based models to alleviate appearance drift issues and enhance the generation of unseen expressions. Specifically, ExpAvatar introduces a Face Priors-Conditioned Diffusion (FPDiff) model to inject 3D face priors into generation models through fine-tuning. Furthermore, a Face Priors-Conditioned Catalyst (FPCatalyst) is employed to enhance the inference efficiency and generation quality. Moreover, we propose a unique confidence-based regularizer function to mitigate the effect of imperfect face-tracking estimates, thereby improving the quality of dynamic neural head avatars. Experimental results demonstrate that ExpAvatar surpasses current state-of-the-art solutions in generating unseen expressions, marking an advancement in the realm of dynamic head avatar synthesis. Code: https://github.com/yuangan/ExpAvatar .
- 7 November 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 21 (11) , 1-21
Open AccessExpAvatar: High-Fidelity Avatar Generation of Unseen Expressions with 3D Face Priors
The reconstruction of dynamic head avatars has gained increasing significance, giving rise to various downstream applications such as visual dubbing and digital human creation. Despite recent advancements, generating novel, unseen expressions for a given identity remains challenging in concurrently achieving (1) accurate expression and consistent appearance and (2) high-quality and realistic faces. This article introduces ExpAvatar, a novel approach crafted to address these challenges. ExpAvatar elaborately leverages the appearance consistency capabilities inherent in 3DMMs-based models along with the robust generalization ability of DDPMs-based models to alleviate appearance drift issues and enhance the generation of unseen expressions. Specifically, ExpAvatar introduces a Face Priors-Conditioned Diffusion (FPDiff) model to inject 3D face priors into generation models through fine-tuning. Furthermore, a Face Priors-Conditioned Catalyst (FPCatalyst) is employed to enhance the inference efficiency and generation quality. Moreover, we propose a unique confidence-based regularizer function to mitigate the effect of imperfect face-tracking estimates, thereby improving the quality of dynamic neural head avatars. Experimental results demonstrate that ExpAvatar surpasses current state-of-the-art solutions in generating unseen expressions, marking an advancement in the realm of dynamic head avatar synthesis. Code: https://github.com/yuangan/ExpAvatar .
- Open Access
- 13 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 1076-1086
ProtChatGPT: Towards Understanding Proteins with Hybrid Representation and Large Language Models
- 13 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 1076-1086
Open AccessProtChatGPT: Towards Understanding Proteins with Hybrid Representation and Large Language Models
- 25 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 6893-6904
CLIP4STR: A Simple Baseline for Scene Text Recognition With Pre-Trained Vision-Language Model
Pre-trained vision-language models (VLMs) are the de-facto foundation models for various downstream tasks. However, scene text recognition methods still prefer backbones pre-trained on a single modality, namely, the visual modality, despite the potential of VLMs to serve as powerful scene text readers. For example, CLIP can robustly identify regular (horizontal) and irregular (rotated, curved, blurred, or occluded) text in images. With such merits, we transform CLIP into a scene text reader and introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP. It has two encoder-decoder branches: a visual branch and a cross-modal branch. The visual branch provides an initial prediction based on the visual feature, and the cross-modal branch refines this prediction by addressing the discrepancy between the visual feature and text semantics. To fully leverage the capabilities of both branches, we design a dual predict-and-refine decoding scheme for inference. We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 13 STR benchmarks. Additionally, a comprehensive empirical study is provided to enhance the understanding of the adaptation of CLIP to STR. We believe our method establishes a simple yet strong baseline for future STR research with VLMs.
- 25 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 6893-6904
CLIP4STR: A Simple Baseline for Scene Text Recognition With Pre-Trained Vision-Language Model
Pre-trained vision-language models (VLMs) are the de-facto foundation models for various downstream tasks. However, scene text recognition methods still prefer backbones pre-trained on a single modality, namely, the visual modality, despite the potential of VLMs to serve as powerful scene text readers. For example, CLIP can robustly identify regular (horizontal) and irregular (rotated, curved, blurred, or occluded) text in images. With such merits, we transform CLIP into a scene text reader and introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP. It has two encoder-decoder branches: a visual branch and a cross-modal branch. The visual branch provides an initial prediction based on the visual feature, and the cross-modal branch refines this prediction by addressing the discrepancy between the visual feature and text semantics. To fully leverage the capabilities of both branches, we design a dual predict-and-refine decoding scheme for inference. We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 13 STR benchmarks. Additionally, a comprehensive empirical study is provided to enhance the understanding of the adaptation of CLIP to STR. We believe our method establishes a simple yet strong baseline for future STR research with VLMs.
- 16 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 6748-6760
Exploiting Unlabeled Videos for Video-Text Retrieval via Pseudo-Supervised Learning
Large-scale pre-trained vision-language models ( e.g ., CLIP) have shown incredible generalization performance in downstream tasks such as video-text retrieval (VTR). Traditional approaches have leveraged CLIP’s robust multi-modal alignment ability for VTR by directly fine-tuning vision and text encoders with clean video-text data. Yet, these techniques rely on carefully annotated video-text pairs, a process that is costly and labor-intensive. In this context, we introduce a new approach, Pseudo-Supervised Selective Contrastive Learning (PS-SCL). PS-SCL minimizes the dependency on manually-labeled text annotations by generating pseudo-supervisions from unlabeled video data for training. We first exploit CLIP’s visual recognition capabilities to generate pseudo-texts automatically. These pseudo-texts contain diverse visual concepts from the video and serve as weak textual guidance. Moreover, we introduce Selective Contrastive Learning (SeLeCT), which prioritizes and selects highly correlated video-text pairs from pseudo-supervised video-text pairs. By doing so, SeLeCT enables more effective multi-modal learning under weak pairing supervision. Experimental results demonstrate that our method outperforms CLIP zero-shot performance by a large margin on multiple video-text retrieval benchmarks, e.g ., 8.2% R@1 for video-to-text on MSRVTT, 12.2% R@1 for video-to-text on DiDeMo, and 10.9% R@1 for video-to-text on ActivityNet, respectively.
- 16 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 6748-6760
Exploiting Unlabeled Videos for Video-Text Retrieval via Pseudo-Supervised Learning
Large-scale pre-trained vision-language models ( e.g ., CLIP) have shown incredible generalization performance in downstream tasks such as video-text retrieval (VTR). Traditional approaches have leveraged CLIP’s robust multi-modal alignment ability for VTR by directly fine-tuning vision and text encoders with clean video-text data. Yet, these techniques rely on carefully annotated video-text pairs, a process that is costly and labor-intensive. In this context, we introduce a new approach, Pseudo-Supervised Selective Contrastive Learning (PS-SCL). PS-SCL minimizes the dependency on manually-labeled text annotations by generating pseudo-supervisions from unlabeled video data for training. We first exploit CLIP’s visual recognition capabilities to generate pseudo-texts automatically. These pseudo-texts contain diverse visual concepts from the video and serve as weak textual guidance. Moreover, we introduce Selective Contrastive Learning (SeLeCT), which prioritizes and selects highly correlated video-text pairs from pseudo-supervised video-text pairs. By doing so, SeLeCT enables more effective multi-modal learning under weak pairing supervision. Experimental results demonstrate that our method outperforms CLIP zero-shot performance by a large margin on multiple video-text retrieval benchmarks, e.g ., 8.2% R@1 for video-to-text on MSRVTT, 12.2% R@1 for video-to-text on DiDeMo, and 10.9% R@1 for video-to-text on ActivityNet, respectively.
- 27 November 2024
- Published by Springer Nature
- p. 73-91
Shape2Scene: 3D Scene Representation Learning Through Pre-training on Shape Data
- 27 November 2024
- Published by Springer Nature
- p. 73-91
Shape2Scene: 3D Scene Representation Learning Through Pre-training on Shape Data
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1952-1960
Neural Interaction Energy for Multi-Agent Trajectory Prediction
Maintaining temporal stability is crucial in multi-agent trajectory prediction. Insufficient regularization to uphold this temporal stability often results in fluctuations in kinematic states, leading to inconsistent predictions and the amplification of errors. In this study, we introduce a framework called Multi-Agent Trajectory prediction via neural interaction Energy (MATE). This framework assesses the interactive motion of agents by employing neural interaction energy, which captures the dynamics of interactions and illustrates their influence on the future trajectories of agents. To bolster temporal stability, we introduce two constraints: inter-agent interaction constraint and intra-agent motion constraint. These constraints work together to ensure temporal stability at both the system and agent levels, effectively mitigating prediction fluctuations inherent in multi-agent systems. Comparative evaluations against previous methods on four diverse datasets, including simulated and real-world scenarios, highlight the superior prediction accuracy and generalization capabilities of our model.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1952-1960
Open AccessNeural Interaction Energy for Multi-Agent Trajectory Prediction
Maintaining temporal stability is crucial in multi-agent trajectory prediction. Insufficient regularization to uphold this temporal stability often results in fluctuations in kinematic states, leading to inconsistent predictions and the amplification of errors. In this study, we introduce a framework called Multi-Agent Trajectory prediction via neural interaction Energy (MATE). This framework assesses the interactive motion of agents by employing neural interaction energy, which captures the dynamics of interactions and illustrates their influence on the future trajectories of agents. To bolster temporal stability, we introduce two constraints: inter-agent interaction constraint and intra-agent motion constraint. These constraints work together to ensure temporal stability at both the system and agent levels, effectively mitigating prediction fluctuations inherent in multi-agent systems. Comparative evaluations against previous methods on four diverse datasets, including simulated and real-world scenarios, highlight the superior prediction accuracy and generalization capabilities of our model.