SWAGS: Sampling Windows Adaptively for Dynamic 3D Gaussian Splatting
Authors:
Preprint
Abstract
Novel view synthesis has shown rapid progress recently, with methods capable of producing evermore photo-realistic results. 3D Gaussian Splatting has emerged as a particularly promising method, producing high-quality renderings of static scenes and enabling interactive viewing at real-time frame rates. However, it is currently limited to static scenes only. In this work, we extend 3D Gaussian Splatting to reconstruct dynamic scenes. We model the dynamics of a scene using a tunable MLP, which learns the deformation field from a canonical space to a set of 3D Gaussians per frame. To disentangle the static and dynamic parts of the scene, we learn a tuneable parameter for each Gaussian, which weighs the respective MLP parameters to focus attention on the dynamic parts. This improves the model's ability to capture dynamics in scenes with an imbalance of static to dynamic regions. To handle scenes of arbitrary length whilst maintaining high rendering quality, we introduce an adaptive window sampling strategy to partition the sequence into windows based on the amount of movement in the sequence. We train a separate dynamic Gaussian Splatting model for each window, allowing the canonical representation to change, thus enabling the reconstruction of scenes with significant geometric or topological changes. Temporal consistency is enforced using a fine-tuning step with self-supervising consistency loss on randomly sampled novel views. As a result, our method produces high-quality renderings of general dynamic scenes with competitive quantitative performance, which can be viewed in real-time with our dynamic interactive viewer.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis2024
- TensoRF: Tensorial Radiance Fields2022
- TAVA: Template-free Animatable Volumetric Actors2022
- FAST-VQA: Efficient End-to-End Video Quality Assessment with Fragment Sampling2022
- Temporal-MPI: Enabling Multi-plane Images for Dynamic Scene Modelling via Temporal Basis Learning2022
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow2020
- KillingFusion: Non-rigid 3D Reconstruction without Correspondences2017
- Structure-from-Motion Revisited2016
- DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time2015
- MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction2014