HoloDiff: Holistic Talking Human Animation Via Latent Diffusion
- 1 July 2026
- journal article
- research article
- Published by Wiley in Computer Animation and Virtual Worlds
- Vol. 37 (4) , e70173
Abstract
Holistic talking human animation aims to synthesize coordinated facial expressions and body poses that are consistent with spoken content and human communication patterns. This task requires both realistic motion generation and accurate temporal alignment between speech and motion, which remain challenging for existing methods that model facial and bodily motion separately or operate in high‐dimensional motion spaces. We present HoloDiff, a latent diffusion framework for holistic talking human animation. Given an input speech signal, HoloDiff encodes facial and bodily dynamics into a compact and structured latent representation that preserves anatomical structure and cross‐part interactions. A speech‐conditioned diffusion process is then applied in this latent space to model long‐range temporal dynamics and generate motion sequences aligned with audio input. By performing diffusion in a structured latent space, HoloDiff reduces modeling complexity while improving temporal coherence and audio–motion alignment. Extensive experiments on the SHOW dataset demonstrate that HoloDiff produces more realistic and expressive talking human animation and achieves better multimodal correspondence between speech and motion compared to baselines.Keywords
This publication has 39 references indexed in Scilit:
- BEAT: A Large-Scale Semantic and Emotional Multi-modal Dataset for Conversational Gestures Synthesis2022
- Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation2022
- Style Transfer for Co-speech Gesture Animation: A Multi-speaker Conditional-Mixture Approach2020
- Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose2020
- Capture, Learning, and Synthesis of 3D Speaking Styles2019
- Expressive Body Capture: 3D Hands, Face, and Body From a Single Image2019
- On the Continuity of Rotation Representations in Neural Networks2019
- Modulation of motor area activity during observation of unnatural body movementsBrain and Cognition, 2012
- Greta. A Believable Embodied Conversational Agent2005
- Synthesizing multimodal utterances for conversational agentsComputer Animation and Virtual Worlds, 2004