DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion
Open Access
- 9 November 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-13
Abstract
Holistic co-speech animation is prone to averaging in both motion representation and speech conditioning. In coordinate-space diffusion, slow body posture, mid-frequency gesture strokes, and fast hand or facial details are entangled in one prediction target, often producing low-variance, over-smoothed motion. Meanwhile, fixed multimodal conditioning can let dense rhythmic and acoustic cues dominate sparse semantic cues, weakening rare content-specific gestures. We present DynaConTalk, a wavelet-constrained diffusion framework for long-form and controllable holistic co-speech motion generation. DynaConTalk performs diffusion in stationary wavelet transform (SWT) coefficient space, where temporally aligned bands separate coarse posture evolution, gesture strokes, and high-frequency expressive details. We further introduce dynamic proposal-consensus conditioning to adaptively balance rhythmic, acoustic, semantic, textual, and identity/activity cues according to the current speech context and denoising state. Finally, to ensure smooth transitions between windows and support flexible user control, we re-noise any reference signal - whether a keypose, a motion clip, or a short boundary condition - to match the current diffusion timestep before injecting it into the denoiser. This unified strategy allows seamless long-form generation, localized editing, and style-driven guidance to be handled within a single coherent process, without task-specific modules or post-hoc stitching. Experiments show that DynaConTalk improves gesture realism, facial accuracy, temporal continuity, and controllable editing over prior holistic co-speech generation methods. Code and data for this paper are at https://zhuyifeiabcd1.github.io/DynaConTalk/.Keywords
This publication has 23 references indexed in Scilit:
- WaveDM: Wavelet-Based Diffusion Models for Image RestorationIEEE Transactions on Multimedia, 2024
- Moving Fast and Slow: Analysis of Representations and Post-Processing in Speech-Driven Automatic Gesture GenerationInternational Journal of Human–Computer Interaction, 2021
- Adversarial gesture generation with realistic gesture phasingComputers & Graphics, 2020
- Style‐Controllable Speech‐Driven Gesture Synthesis Using Normalising FlowsComputer Graphics Forum, 2020
- Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots2019
- Investigating the use of recurrent motion modelling for speech gesture generation2018
- Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network2018
- Phase-functioned neural networks for character controlACM Transactions on Graphics, 2017
- Aspects of rhythm in gesture and speechGesture, 2007
- MPML: a markup language for controlling the behavior of life-like charactersJournal of Visual Languages & Computing, 2004