GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation
Preprint
Abstract
Creating 4D fields of Gaussian Splatting from images or videos is a challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the input videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to be handled by existing methods. The common color drifting issue that happens in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality on extensive experiments demonstrates our method's effectiveness. Quantitative and qualitative evaluations show that our method achieves state-of-the-art results on both tasks of 4D generation and 4D novel view synthesis. Project page: https://zerg-overmind.github.io/GaussianFlow.github.io/Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM2024
- Wonder3D: Single Image to 3D Using Cross-Domain Diffusion2024
- Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models2024
- 4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling2024
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering2024
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion2024
- Gaussian Splatting SLAM2024
- Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis2024
- DreamBooth3D: Subject-Driven Text-to-3D Generation2023
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020