SingingHead: A Large-Scale 4D Dataset for Singing Head Animation
Top Cited Papers
- 20 October 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28 (15209210) , 700-714
Abstract
Singing, as a common facial movement second only to talking, can be regarded as a universal language across ethnicities and cultures, plays an important role in emotional communication, art, and entertainment. However, it is often overlooked in the field of audio-driven 3D facial animation due to the lack of singing head datasets and the domain gap between singing and talking in rhythm and amplitude. To this end, we collect a large-scale high-quality multi-modal singing head dataset, SingingHead, which consists of more than 27 hours of synchronized singing video, 3D facial motion, singing audio, and background music from 76 individuals and 8 types of music. Along with the SingingHead dataset, we benchmark existing audio-driven 3D facial animation methods and 2D talking head methods on the singing task. Existing 3D facial animation methods and 2D talking head methods fail to produce satisfactory singing results. Focusing on the 3D singing head animation, we first utilize the proposed singing-specific dataset to retrain the 3D facial animation methods, resulting in substantial performance improvements. Besides, considering the absence of background music and the slow generation speed of existing methods, we propose a simple but efficient non-autoregressive VAE-based framework with background music as an input signal to generate diverse and accurate 3D singing facial motions in real time. Extensive experiments demonstrate the significance of the SingingHead dataset in promoting the development of singing head animation. The dataset is released for research purposes at: https://wsj-sjtu.github.io/SingingHead/.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62225112)
- National Natural Science Foundation of China (62132006)
- National Natural Science Foundation of China (62101326)
- National Natural Science Foundation of China (U24A20220)
- National Natural Science Foundation of China (62571324)
This publication has 80 references indexed in Scilit:
- UniTalker: Scaling up Audio-Driven 3D Facial Animation Through A Unified Model2024
- EMO: Emote Portrait Alive Generating Expressive Portrait Videos with Audio2Video Diffusion Model Under Weak Conditions2024
- MusicFace: Music-driven expressive singing face synthesisComputational Visual Media, 2024
- Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation2022
- MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation2020
- Lip Movements Generation at a Glance2018
- Lip Reading Sentences in the Wild2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Out of Time: Automated Lip Sync in the Wild2017
- Design, analysis and experimental evaluation of block based transformation in MFCC computation for speaker recognitionSpeech Communication, 2012