4D Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes
Authors:
- Baoquan ChenBCBaoquan Chen
Preprint
Abstract
We consider the problem of novel view synthesis (NVS) for dynamic scenes. Recent neural approaches have accomplished exceptional NVS results for static 3D scenes, but extensions to 4D time-varying scenes remain non-trivial. Prior efforts often encode dynamics by learning a canonical space plus implicit or explicit deformation fields, which struggle in challenging scenarios like sudden movements or capturing high-fidelity renderings. In this paper, we introduce 4D Gaussian Splatting (4DGS), a novel method that represents dynamic scenes with anisotropic 4D XYZT Gaussians, inspired by the success of 3D Gaussian Splatting in static scenes. We model dynamics at each timestamp by temporally slicing the 4D Gaussians, which naturally compose dynamic 3D Gaussians and can be seamlessly projected into images. As an explicit spatial-temporal representation, 4DGS demonstrates powerful capabilities for modeling complicated dynamics and fine details, especially for scenes with abrupt motions. We further implement our temporal slicing and splatting techniques in a highly optimized CUDA acceleration framework, achieving real-time inference rendering speeds of up to 277 FPS on an RTX 3090 GPU and 583 FPS on an RTX 4090 GPU. Rigorous evaluations on scenes with diverse motions showcase the superior efficiency and effectiveness of 4DGS, which consistently outperforms existing methods both quantitatively and qualitatively.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction2024
- Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis2024
- NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance FieldsIEEE Transactions on Visualization and Computer Graphics, 2023
- ColMap: A memory-efficient occupancy grid mapping frameworkRobotics and Autonomous Systems, 2021
- Deep blending for free-viewpoint image-based renderingACM Transactions on Graphics, 2018
- Stereo magnificationACM Transactions on Graphics, 2018
- Soft 3D reconstruction for view synthesisACM Transactions on Graphics, 2017
- High-quality streamable free-viewpoint videoACM Transactions on Graphics, 2015
- Temporally coherent completion of dynamic shapesACM Transactions on Graphics, 2012
- Virtualized reality: constructing virtual worlds from real scenesIEEE MultiMedia, 1997