Mingyi Shi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Mingyi Shi has a total of 20 co-authors. Here are the top co-authors by number of shared publications:
- Taku Komura
University of Hong Kong, Hong Kong, China
5 shared publications - Xuelin Chen
Tencent AI Lab, China
2 shared publications - Dafei Qin
University of Hong Kong, Hong Kong, China
1 shared publication - Huancheng Lin
The University of Hong Kong, Hong Kong, Hong Kong
1 shared publication - Jungdam Won
Seoul National University, Seoul, Republic of Korea
1 shared publication - Junichi Yamagishi
National Institute of Informatics, Tokyo, Japan
1 shared publication
Latest Publications
- Open Access
- 30 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Neural Codec for Skeletal Animation Compression
- 30 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Open AccessNeural Codec for Skeletal Animation Compression
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-13
DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion
Holistic co-speech animation is prone to averaging in both motion representation and speech conditioning. In coordinate-space diffusion, slow body posture, mid-frequency gesture strokes, and fast hand or facial details are entangled in one prediction target, often producing low-variance, over-smoothed motion. Meanwhile, fixed multimodal conditioning can let dense rhythmic and acoustic cues dominate sparse semantic cues, weakening rare content-specific gestures. We present DynaConTalk, a wavelet-constrained diffusion framework for long-form and controllable holistic co-speech motion generation. DynaConTalk performs diffusion in stationary wavelet transform (SWT) coefficient space, where temporally aligned bands separate coarse posture evolution, gesture strokes, and high-frequency expressive details. We further introduce dynamic proposal-consensus conditioning to adaptively balance rhythmic, acoustic, semantic, textual, and identity/activity cues according to the current speech context and denoising state. Finally, to ensure smooth transitions between windows and support flexible user control, we re-noise any reference signal - whether a keypose, a motion clip, or a short boundary condition - to match the current diffusion timestep before injecting it into the denoiser. This unified strategy allows seamless long-form generation, localized editing, and style-driven guidance to be handled within a single coherent process, without task-specific modules or post-hoc stitching. Experiments show that DynaConTalk improves gesture realism, facial accuracy, temporal continuity, and controllable editing over prior holistic co-speech generation methods. Code and data for this paper are at https://zhuyifeiabcd1.github.io/DynaConTalk/.
- 9 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-13
Open AccessDynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion
Holistic co-speech animation is prone to averaging in both motion representation and speech conditioning. In coordinate-space diffusion, slow body posture, mid-frequency gesture strokes, and fast hand or facial details are entangled in one prediction target, often producing low-variance, over-smoothed motion. Meanwhile, fixed multimodal conditioning can let dense rhythmic and acoustic cues dominate sparse semantic cues, weakening rare content-specific gestures. We present DynaConTalk, a wavelet-constrained diffusion framework for long-form and controllable holistic co-speech motion generation. DynaConTalk performs diffusion in stationary wavelet transform (SWT) coefficient space, where temporally aligned bands separate coarse posture evolution, gesture strokes, and high-frequency expressive details. We further introduce dynamic proposal-consensus conditioning to adaptively balance rhythmic, acoustic, semantic, textual, and identity/activity cues according to the current speech context and denoising state. Finally, to ensure smooth transitions between windows and support flexible user control, we re-noise any reference signal - whether a keypose, a motion clip, or a short boundary condition - to match the current diffusion timestep before injecting it into the denoiser. This unified strategy allows seamless long-form generation, localized editing, and style-driven guidance to be handled within a single coherent process, without task-specific modules or post-hoc stitching. Experiments show that DynaConTalk improves gesture realism, facial accuracy, temporal continuity, and controllable editing over prior holistic co-speech generation methods. Code and data for this paper are at https://zhuyifeiabcd1.github.io/DynaConTalk/.
- Open Access
- 8 August 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Computer Graphics and Interactive Techniques
- Vol. 8 (4) , 1-27
InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios 53
We address the problem of accurate capture of interactive behaviors between two people in daily scenarios. Most previous works either only consider one person or solely focus on conversational gestures of two people, assuming the body orientation and/or position of each actor are constant or barely change over each interaction. In contrast, we propose to simultaneously model two people’s activities, and target objective-driven, dynamic, and semantically consistent interactions which often span longer duration and cover bigger space. To this end, we capture a new multi-modal dataset dubbed InterAct, which is composed of 241 motion sequences where two people perform a realistic and coherent scenario for one minute or longer over a complete interaction. For each sequence, two actors are assigned different roles and emotion labels, and collaborate to finish one task or conduct a common interaction activity. The audios, body motions, and facial expressions of both persons are captured. InterAct contains diverse and complex motions of individuals and interesting and relatively long-term interaction patterns barely seen before. We also demonstrate a simple yet effective diffusion-based method that estimates interactive face expressions and body motions of two people from speech inputs. Our method regresses the body motions in a hierarchical manner, and we also propose a novel fine-tuning mechanism to improve the lip accuracy of facial expressions. To facilitate further research, the data and code will be made public.
- 8 August 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Computer Graphics and Interactive Techniques
- Vol. 8 (4) , 1-27
Open AccessInterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios 53
We address the problem of accurate capture of interactive behaviors between two people in daily scenarios. Most previous works either only consider one person or solely focus on conversational gestures of two people, assuming the body orientation and/or position of each actor are constant or barely change over each interaction. In contrast, we propose to simultaneously model two people’s activities, and target objective-driven, dynamic, and semantically consistent interactions which often span longer duration and cover bigger space. To this end, we capture a new multi-modal dataset dubbed InterAct, which is composed of 241 motion sequences where two people perform a realistic and coherent scenario for one minute or longer over a complete interaction. For each sequence, two actors are assigned different roles and emotion labels, and collaborate to finish one task or conduct a common interaction activity. The audios, body motions, and facial expressions of both persons are captured. InterAct contains diverse and complex motions of individuals and interesting and relatively long-term interaction patterns barely seen before. We also demonstrate a simple yet effective diffusion-based method that estimates interactive face expressions and body motions of two people from speech inputs. Our method regresses the body motions in a hierarchical manner, and we also propose a novel fine-tuning mechanism to improve the lip accuracy of facial expressions. To facilitate further research, the data and code will be made public.
- Open Access
- 13 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-10
Taming Diffusion Probabilistic Models for Character Control
We present a novel character control framework that effectively utilizes motion diffusion probabilistic models to generate high-quality and diverse character animations, responding in real-time to a variety of dynamic user-supplied control signals. At the heart of our method lies a transformer-based Conditional Autoregressive Motion Diffusion Model (CAMDM), which takes as input the character’s historical motion and can generate a range of diverse potential future motions conditioned on high-level, coarse user control. To meet the demands for diversity, controllability, and computational efficiency required by a real-time controller, we incorporate several key algorithmic designs. These include separate condition tokenization, classifier-free guidance on past motion, and heuristic future trajectory extension, all designed to address the challenges associated with taming motion diffusion probabilistic models for character control. As a result, our work represents the first model that enables real-time generation of high-quality, diverse character animations based on user interactive control, supporting animating the character in multiple styles with a single unified model. We evaluate our method on a diverse set of locomotion skills, demonstrating the merits of our method over existing character controllers. The code and model are available at https://aiganimation.github.io/CAMDM/.
- 13 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-10
Open AccessTaming Diffusion Probabilistic Models for Character Control
We present a novel character control framework that effectively utilizes motion diffusion probabilistic models to generate high-quality and diverse character animations, responding in real-time to a variety of dynamic user-supplied control signals. At the heart of our method lies a transformer-based Conditional Autoregressive Motion Diffusion Model (CAMDM), which takes as input the character’s historical motion and can generate a range of diverse potential future motions conditioned on high-level, coarse user control. To meet the demands for diversity, controllability, and computational efficiency required by a real-time controller, we incorporate several key algorithmic designs. These include separate condition tokenization, classifier-free guidance on past motion, and heuristic future trajectory extension, all designed to address the challenges associated with taming motion diffusion probabilistic models for character control. As a result, our work represents the first model that enables real-time generation of high-quality, diverse character animations based on user interactive control, supporting animating the character in multiple styles with a single unified model. We evaluate our method on a diverse set of locomotion skills, demonstrating the merits of our method over existing character controllers. The code and model are available at https://aiganimation.github.io/CAMDM/.
- 1 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14679-14691
PhaseMP: Robust 3D Pose Estimation via Phase-conditioned Human Motion Prior
We present a novel motion prior, called PhaseMP, modeling a probability distribution on pose transitions conditioned by a frequency domain feature extracted from a periodic autoencoder. The phase feature further enforces the pose transitions to be unidirectional (i.e. no backward movement in time), from which more stable and natural motions can be generated. Specifically, our motion prior can be useful for accurately estimating 3D human motions in the presence of challenging input data, including long periods of spatial and temporal occlusion, as well as noisy sensor measurements. Through a comprehensive evaluation, we demonstrate the efficacy of our novel motion prior, showcasing its superiority over existing state-of-the-art methods by a significant margin across various applications, including video-to-motion and motion estimation from sparse sensor data, and etc.
- 1 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14679-14691
PhaseMP: Robust 3D Pose Estimation via Phase-conditioned Human Motion Prior
We present a novel motion prior, called PhaseMP, modeling a probability distribution on pose transitions conditioned by a frequency domain feature extracted from a periodic autoencoder. The phase feature further enforces the pose transitions to be unidirectional (i.e. no backward movement in time), from which more stable and natural motions can be generated. Specifically, our motion prior can be useful for accurately estimating 3D human motions in the presence of challenging input data, including long periods of spatial and temporal occlusion, as well as noisy sensor measurements. Through a comprehensive evaluation, we demonstrate the efficacy of our novel motion prior, showcasing its superiority over existing state-of-the-art methods by a significant margin across various applications, including video-to-motion and motion estimation from sparse sensor data, and etc.