Audio-driven Gesture Generation via Deviation Feature in the Latent Space
Preprint
Abstract
Gestures are essential for enhancing co-speech communication, offering visual emphasis and complementing verbal interactions. While prior work has concentrated on point-level motion or fully supervised data-driven methods, we focus on co-speech gestures, advocating for weakly supervised learning and pixel-level motion deviations. We introduce a weakly supervised framework that learns latent representation deviations, tailored for co-speech gesture video generation. Our approach employs a diffusion model to integrate latent motion features, enabling more precise and nuanced gesture representation. By leveraging weakly supervised deviations in latent space, we effectively generate hand gestures and mouth movements, crucial for realistic video production. Experiments show our method significantly improves video quality, surpassing current state-of-the-art techniques.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis2025
- Disco: Disentangled Control for Realistic Human Dance Generation2024
- VBench: Comprehensive Benchmark Suite for Video Generative Models2024
- MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation2023
- Thin-Plate Spline Motion Model for Image Animation2022
- Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory2022
- Motion Representations for Articulated Animation2021
- Learning Individual Styles of Conversational Gesture2019
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- Learning from Gesture: How Our Hands Change Our MindsEducational Psychology Review, 2015