SpeechAct: Towards Generating Whole-Body Motion From Speech
- 13 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (10) , 6737-6750
Abstract
Whole-body motion generation from speech audio is crucial for computer graphics and immersive VR/AR. Prior methods struggle to produce natural and diverse whole-body motions from speech. In this paper, we introduce a novel method, named SpeechAct, based on a hybrid point representation and contrastive motion learning to boost realism and diversity in motion generation. Our hybrid point representation leverages the advantages of keypoint representation and surface points of 3D body model, which is easy to learn and helps to achieve smooth and natural motion generation from speech audio. We design a VQ-VAE to learn a motion codebook using our hybrid presentation, and then regress the motion from the input audio using a translation model. To boost diversity in motion generation, we propose a contrastive motion learning method according to the intuitive idea that the generated motion should be different from the motions of other audios and other speakers. We collect negative samples from other audio inputs and other speakers using our translation model. With these negative samples, we pull the current motion away from them using a contrastive loss to produce more distinctive representations. In addition, we compose a face generator to generate deterministic face motion due to the strong connection between the face movements and the speech audio. Experimental results validate the superior performance of our model. The code is available at http://cic.tju.edu.cn/faculty/likun/projects/SpeechAct/index.html.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2023YFC3082100)
- National Natural Science Foundation of China (62122058, 62171317)
- Science Fund for Distinguished Young Scholars of Tianjin Municipality (22JCJQJC00040)
This publication has 61 references indexed in Scilit:
- Expressive Whole-Body 3D Gaussian Avatar2024
- MILI: Multi-person inference from a low-resolution imageFundamental Research, 2023
- COUCH: Towards Controllable Human-Chair Interactions2022
- Style Transfer for Co-speech Gesture Animation: A Multi-speaker Conditional-Mixture Approach2020
- Contrastive Learning for Unpaired Image-to-Image Translation2020
- Long-Term Human Motion Prediction with Scene Context2020
- Info3D: Representation Learning on 3D Objects Using Mutual Information Maximization and Contrastive Learning2020
- Generating 3D Faces Using Convolutional Mesh Autoencoders2018
- Creating a Gesture-Speech Dataset for Speech-Based Automatic Gesture Generation2017
- Synthesizing multimodal utterances for conversational agentsComputer Animation and Virtual Worlds, 2004