OT-Talk: Animating 3D Talking Head with Optimal Transportation
Authors:
- Xinmu WangXWXinmu Wang
- Xiang GaoXGXiang Gao
- Xiyun SongXSXiyun Song
- Heather YuHYHeather Yu
- Zongfang LinZLZongfang Lin
- Liang PengLPLiang Peng
- Xianfeng GuXGXianfeng Gu
Preprint
Abstract
Animating 3D head meshes using audio inputs has significant applications in AR/VR, gaming, and entertainment through 3D avatars. However, bridging the modality gap between speech signals and facial dynamics remains a challenge, often resulting in incorrect lip syncing and unnatural facial movements. To address this, we propose OT-Talk, the first approach to leverage optimal transportation to optimize the learning model in talking head animation. Building on existing learning frameworks, we utilize a pre-trained Hubert model to extract audio features and a transformer model to process temporal sequences. Unlike previous methods that focus solely on vertex coordinates or displacements, we introduce Chebyshev Graph Convolution to extract geometric features from triangulated meshes. To measure mesh dissimilarities, we go beyond traditional mesh reconstruction errors and velocity differences between adjacent frames. Instead, we represent meshes as probability measures and approximate their surfaces. This allows us to leverage the sliced Wasserstein distance for modeling mesh variations. This approach facilitates the learning of smooth and accurate facial motions, resulting in coherent and natural facial animations. Our experiments on two public audio-mesh datasets demonstrate that our method outperforms state-of-the-art techniques both quantitatively and qualitatively in terms of mesh reconstruction accuracy and temporal alignment. In addition, we conducted a user perception study with 20 volunteers to further assess the effectiveness of our approach.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Pose-Aware 3D Talking Face Synthesis Using Geometry-Guided Audio-Vertices AttentionIEEE Transactions on Visualization and Computer Graphics, 2024
- Geometry-Guided Dense Perspective Network for Speech-Driven Facial AnimationIEEE Transactions on Visualization and Computer Graphics, 2021
- On parameter estimation with the Wasserstein distanceInformation and Inference: A Journal of the IMA, 2019
- Realistic Speech-Driven Facial Animation with GANsInternational Journal of Computer Vision, 2019
- Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation2019
- Generating 3D Faces using Multi‐column Graph Convolutional NetworksComputer Graphics Forum, 2019
- Audio-driven facial animation by joint end-to-end learning of pose and emotionACM Transactions on Graphics, 2017
- Sliced and Radon Wasserstein Barycenters of MeasuresJournal of Mathematical Imaging and Vision, 2014
- The Varifold Representation of Nonoriented Shapes for Diffeomorphic RegistrationSIAM Journal on Imaging Sciences, 2013
- A 3-D Audio-Visual Corpus of Affective CommunicationIEEE Transactions on Multimedia, 2010