HP-Gaussian: Head Prior-Guided Gaussian Splatting for Personalized Talking Head Synthesis From Few-Second Video
- 2 June 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 5880-5893
Abstract
Gaussian Splatting-based talking head synthesis has made significant progress in recent years, yet existing methods often struggle with generalization beyond specific training identity. In this paper, we propose Head Prior guided Gaussian Splatting for personalized talking head synthesis (HP-Gaussian) that can generalize to new identities with only few training data. Unlike traditional optimization-based Gaussian Splatting methods, our approach directly predicts Gaussian parameters from multi-modal inputs, including audio and visual cues. This feed-forward design enables multiple identities pre-training, allowing the model to learn shared head priors from large-scale datasets, while supporting flexible speaker-specific adaptation. To further enhance Gaussian feature learning, we introduce a Spatial Gaussian Transformer that captures correlations among neighboring Gaussians, improving parameter estimation accuracy. Additionally, recognizing the critical importance of personalized speaking styles in the synthesis of high-quality talking videos, a two-stage training strategy is implemented. A base model is initially trained across diverse identities to establish the foundational head prior knowledge. Subsequently, we introduce the short-video personalized adaptation phase for more realistic customized talking video generation. Extensive experiments demonstrate that our HP-Gaussian can synthesize high-fidelity and personalized talking videos with remarkably few training examples, setting a new benchmark for efficiency and quality in talking head synthesis. We highly recommend viewing our demonstration video at https://youtu.be/rAWqkg00WBM1 for intuitive visual comparisons and qualitative results.Keywords
Funding Information
- National Research Foundation, Singapore under NRF Medium Sized Centre Scheme
- National Natural Science Foundation of China (62125603)
- National Natural Science Foundation of China (62336004)
- National Natural Science Foundation of China (62321005)
- National Natural Science Foundation of China (62441616)
This publication has 63 references indexed in Scilit:
- Real-Time Neural Radiance Talking Portrait Synthesis via Audio-Spatial DecompositionInternational Journal of Computer Vision, 2025
- TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting2024
- MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images2024
- Scalable watermarking for identifying large language model outputsNature, 2024
- EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head2024
- Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis2022
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Neural Voice Puppetry: Audio-Driven Facial Reenactment2020
- Realistic Speech-Driven Facial Animation with GANsInternational Journal of Computer Vision, 2019
- Out of Time: Automated Lip Sync in the Wild2017