Xiaoyu Qin
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Xiaoyu Qin has a total of 50 co-authors. Here are the top co-authors by number of shared publications:
- Jia Jia
Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China
6 shared publications - Zehai Niu
School of Artificial Intelligence, Shenzhen University, Shenzhen, China
6 shared publications - Jian Xue
University of Chinese Academy of Sciences, Beijing, China
5 shared publications - Jinbao Wang
Shenzhen University
5 shared publications - Ke Lu
School of Software, Henan University, Kaifeng, China
5 shared publications - Zixuan Wang
Tsinghua University, Beijing, China
5 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 4077-4086
Question-in-the-Loop: Interleaved Query Tokens as Visual Summarizers for Video LLMs
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 4077-4086
Open AccessQuestion-in-the-Loop: Interleaved Query Tokens as Visual Summarizers for Video LLMs
- Open Access
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-11
VFAvatar: Feed-Forward 3D Avatar Reconstruction from Casual Image Collections
The growing demand for personalized 3D avatars calls for efficient reconstruction methods from casual photos. This task remains challenging due to unconstrained viewpoints, partial body visibility, and temporal variations across input images. While some previous methods circumvent these difficulties by adopting generative approaches like score distillation, they struggle to preserve authentic appearance details from source images. To address these limitations, we introduce Visual-Fusion-Avatar (VFAvatar), a novel feed-forward framework that reconstructs 3D avatars by fusing visual cues in just a few seconds. VFAvatar couples a pose-free reconstruction foundation model with a pretrained human generation prior in a mutually reinforcing manner. And we propose a visibility-aware, view-attentive residual aggregation mechanism that routes and fuses per-view updates, allowing partial observations from different images to be assembled into a single coherent avatar. Experiments demonstrate that VFAvatar significantly outperforms state-of-the-art methods in both reconstruction fidelity and efficiency, while enabling shape and pose manipulation. Code is available on: https://github.com/huangshuo200823/VFAvatar.
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Open AccessVFAvatar: Feed-Forward 3D Avatar Reconstruction from Casual Image Collections
The growing demand for personalized 3D avatars calls for efficient reconstruction methods from casual photos. This task remains challenging due to unconstrained viewpoints, partial body visibility, and temporal variations across input images. While some previous methods circumvent these difficulties by adopting generative approaches like score distillation, they struggle to preserve authentic appearance details from source images. To address these limitations, we introduce Visual-Fusion-Avatar (VFAvatar), a novel feed-forward framework that reconstructs 3D avatars by fusing visual cues in just a few seconds. VFAvatar couples a pose-free reconstruction foundation model with a pretrained human generation prior in a mutually reinforcing manner. And we propose a visibility-aware, view-attentive residual aggregation mechanism that routes and fuses per-view updates, allowing partial observations from different images to be assembled into a single coherent avatar. Experiments demonstrate that VFAvatar significantly outperforms state-of-the-art methods in both reconstruction fidelity and efficiency, while enabling shape and pose manipulation. Code is available on: https://github.com/huangshuo200823/VFAvatar.
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 6720-6729
HarmoniVox: Painting Voices to Match the Avatar's Soul
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 6720-6729
Open AccessHarmoniVox: Painting Voices to Match the Avatar's Soul
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 10239-10247
DEPO: Enhancing E-commerce Image Background Generation with Short Trajectory Direct Expected Preference Optimization
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 10239-10247
Open AccessDEPO: Enhancing E-commerce Image Background Generation with Short Trajectory Direct Expected Preference Optimization
- Open Access
V-CASS: Vision-context-aware Expressive Speech Synthesis for Enhancing User Understanding of Videos
Automatic video commentary systems are widely used on multimedia social media platforms to extract factual information about video content. However, current systems may overlook essential para-linguistic cues, including emotion and attitude, which are critical for fully conveying the meaning of visual content. The absence of these cues can limit user understanding or, in some cases, distort the video's original intent. Expressive speech effectively conveys these cues and enhances the user's comprehension of videos. Building on these insights, this paper explores the usage of vision-context-aware expressive speech in enhancing users' understanding of videos in video commentary systems. Firstly, our formatting study indicates that semantic-only speech can lead to ambiguity, and misaligned emotions between speech and visuals may distort content interpretation. To address this, we propose a method called vision-context-aware speech synthesis (V-CASS). It analyzes para-linguistic cues from visuals using a vision-language model and leverages a knowledge-infused language model to guide the expressive speech model in generating context-aligned speech. User studies show that V-CASS enhances emotional and attitudinal resonance, as well as user audio-visual understanding and engagement, with 74.68% of participants preferring the system. Finally, we explore the potential of our method in helping blind and low-vision users navigate web videos, improving universal accessibility.
Open AccessV-CASS: Vision-context-aware Expressive Speech Synthesis for Enhancing User Understanding of Videos
Automatic video commentary systems are widely used on multimedia social media platforms to extract factual information about video content. However, current systems may overlook essential para-linguistic cues, including emotion and attitude, which are critical for fully conveying the meaning of visual content. The absence of these cues can limit user understanding or, in some cases, distort the video's original intent. Expressive speech effectively conveys these cues and enhances the user's comprehension of videos. Building on these insights, this paper explores the usage of vision-context-aware expressive speech in enhancing users' understanding of videos in video commentary systems. Firstly, our formatting study indicates that semantic-only speech can lead to ambiguity, and misaligned emotions between speech and visuals may distort content interpretation. To address this, we propose a method called vision-context-aware speech synthesis (V-CASS). It analyzes para-linguistic cues from visuals using a vision-language model and leverages a knowledge-infused language model to guide the expressive speech model in generating context-aligned speech. User studies show that V-CASS enhances emotional and attitudinal resonance, as well as user audio-visual understanding and engagement, with 74.68% of participants preferring the system. Finally, we explore the potential of our method in helping blind and low-vision users navigate web videos, improving universal accessibility.
- 12 May 2025
- Published by Springer Nature
- p. 274-289
FlexControl: Flexible and Efficient Full-Body Controllable Text-to-Motion Generation
- 12 May 2025
- Published by Springer Nature
- p. 274-289
FlexControl: Flexible and Efficient Full-Body Controllable Text-to-Motion Generation
- 12 May 2025
- Published by Springer Nature
- p. 307-323
Motion Reconstruction via Human Anatomy Diffusion from Sparse Tracking
- 12 May 2025
- Published by Springer Nature
- p. 307-323
Motion Reconstruction via Human Anatomy Diffusion from Sparse Tracking
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4024-4033
Realistic Full-Body Motion Generation from Sparse Tracking with State Space Model
In the domain of generative multimedia and interactive experiences, generating realistic and accurate full-body poses from sparse tracking is crucial for many real-world applications, while achieving sequence modeling and efficient motion generation remains challenging. Recently, state space models (SSMs) with efficient hardware-aware designs (i.e., Mamba) have shown great potential for sequence modeling, particularly in temporal contexts. However, processing motion data is still challenging for SSMs. Specifically, the sparsity of input conditions makes motion generation an ill-posed problem. Moreover, the complex structure of the human body further complicates this task. To address these issues, we present Motion Mamba Diffusion (MMD), a novel conditional diffusion model, which effectively utilizes the sequence modeling capability of SSMs and the robust generation ability of diffusion models to track full-body poses accurately. In particular, we design a bidirectional Temporal Mamba Module (TMM) to model motion sequence. Additionally, a Spatial Mamba Module (SMM) is further proposed for feature enhancement within a single frame. Extensive experiments on the large motion capture dataset (AMASS) demonstrate that our proposed approach outperforms the latest methods in terms of accuracy and smoothness, thus providing a crucial advancement for creating realistic virtual avatars in various applications.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4024-4033
Open AccessRealistic Full-Body Motion Generation from Sparse Tracking with State Space Model
In the domain of generative multimedia and interactive experiences, generating realistic and accurate full-body poses from sparse tracking is crucial for many real-world applications, while achieving sequence modeling and efficient motion generation remains challenging. Recently, state space models (SSMs) with efficient hardware-aware designs (i.e., Mamba) have shown great potential for sequence modeling, particularly in temporal contexts. However, processing motion data is still challenging for SSMs. Specifically, the sparsity of input conditions makes motion generation an ill-posed problem. Moreover, the complex structure of the human body further complicates this task. To address these issues, we present Motion Mamba Diffusion (MMD), a novel conditional diffusion model, which effectively utilizes the sequence modeling capability of SSMs and the robust generation ability of diffusion models to track full-body poses accurately. In particular, we design a bidirectional Temporal Mamba Module (TMM) to model motion sequence. Additionally, a Spatial Mamba Module (SMM) is further proposed for feature enhancement within a single frame. Extensive experiments on the large motion capture dataset (AMASS) demonstrate that our proposed approach outperforms the latest methods in terms of accuracy and smoothness, thus providing a crucial advancement for creating realistic virtual avatars in various applications.
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 10200-10209
DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis
Synthesizing camera movements from music and dance is highly challenging due to the contradicting requirements and complexities of dance cinematography. Unlike human movements, which are always continuous, dance camera movements involve both continuous sequences of variable lengths and sudden drastic changes to simulate the switching of multiple cameras. However, in previous works, every camera frame is equally treated and this causes jittering and unavoidable smoothing in post-processing. To solve these problems, we propose to integrate animator dance cinematography knowledge by formulating this task as a three-stage process: keyframe detection, keyframe synthesis, and tween function prediction. Following this formulation, we design a novel end-to-end dance camera synthesis framework DanceCamAnimator, which imitates human animation procedures and shows powerful keyframe-based controllability with variable lengths. Extensive experiments on the DCM dataset demonstrate that our method surpasses previous baselines quantitatively and qualitatively. Code will be available at https://github.com/Carmenw1203/DanceCamAnimator-Official.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 10200-10209
Open AccessDanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis
Synthesizing camera movements from music and dance is highly challenging due to the contradicting requirements and complexities of dance cinematography. Unlike human movements, which are always continuous, dance camera movements involve both continuous sequences of variable lengths and sudden drastic changes to simulate the switching of multiple cameras. However, in previous works, every camera frame is equally treated and this causes jittering and unavoidable smoothing in post-processing. To solve these problems, we propose to integrate animator dance cinematography knowledge by formulating this task as a three-stage process: keyframe detection, keyframe synthesis, and tween function prediction. Following this formulation, we design a novel end-to-end dance camera synthesis framework DanceCamAnimator, which imitates human animation procedures and shows powerful keyframe-based controllability with variable lengths. Extensive experiments on the DCM dataset demonstrate that our method surpasses previous baselines quantitatively and qualitatively. Code will be available at https://github.com/Carmenw1203/DanceCamAnimator-Official.
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 554-563
VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling
Recent AIGC systems possess the capability to generate digital multimedia content based on human language instructions, such as text, image and video. However, when it comes to speech, existing methods related to human instruction-to-speech generation exhibit two limitations. Firstly, they require the division of inputs into content prompt (transcript) and description prompt (style and speaker), instead of directly supporting human instruction. This division is less natural in form and does not align with other AIGC models. Secondly, the practice of utilizing an independent description prompt to model speech style, without considering the transcript content, restricts the ability to control speech at a fine-grained level. To address these limitations, we propose VoxInstruct, a novel unified multilingual codec language modeling framework that extends traditional text-to-speech tasks into a general human instruction-to-speech task. Our approach enhances the expressiveness of human instruction-guided speech generation and aligns the speech generation paradigm with other modalities. To enable the model to automatically extract the content of synthesized speech from raw text instructions, we introduce speech semantic tokens as an intermediate representation for instruction-to-content guidance. We also incorporate multiple Classifier-Free Guidance (CFG) strategies into our codec language model, which strengthens the generated speech following human instructions. Furthermore, our model architecture and training strategies allow for the simultaneous support of combining speech prompt and descriptive human instruction for expressive speech synthesis, which is a first-of-its-kind attempt. Codes, models and demos are at: https://github.com/thuhcsi/VoxInstruct.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 554-563
Open AccessVoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling
Recent AIGC systems possess the capability to generate digital multimedia content based on human language instructions, such as text, image and video. However, when it comes to speech, existing methods related to human instruction-to-speech generation exhibit two limitations. Firstly, they require the division of inputs into content prompt (transcript) and description prompt (style and speaker), instead of directly supporting human instruction. This division is less natural in form and does not align with other AIGC models. Secondly, the practice of utilizing an independent description prompt to model speech style, without considering the transcript content, restricts the ability to control speech at a fine-grained level. To address these limitations, we propose VoxInstruct, a novel unified multilingual codec language modeling framework that extends traditional text-to-speech tasks into a general human instruction-to-speech task. Our approach enhances the expressiveness of human instruction-guided speech generation and aligns the speech generation paradigm with other modalities. To enable the model to automatically extract the content of synthesized speech from raw text instructions, we introduce speech semantic tokens as an intermediate representation for instruction-to-content guidance. We also incorporate multiple Classifier-Free Guidance (CFG) strategies into our codec language model, which strengthens the generated speech following human instructions. Furthermore, our model architecture and training strategies allow for the simultaneous support of combining speech prompt and descriptive human instruction for expressive speech synthesis, which is a first-of-its-kind attempt. Codes, models and demos are at: https://github.com/thuhcsi/VoxInstruct.