FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally
Preprint
Abstract
This study addresses the challenge of accurately segmenting 3D Gaussian Splatting from 2D masks. Conventional methods often rely on iterative gradient descent to assign each Gaussian a unique label, leading to lengthy optimization and sub-optimal solutions. Instead, we propose a straightforward yet globally optimal solver for 3D-GS segmentation. The core insight of our method is that, with a reconstructed 3D-GS scene, the rendering of the 2D masks is essentially a linear function with respect to the labels of each Gaussian. As such, the optimal label assignment can be solved via linear programming in closed form. This solution capitalizes on the alpha blending characteristic of the splatting process for single step optimization. By incorporating the background bias in our objective function, our method shows superior robustness in 3D segmentation against noises. Remarkably, our optimization completes within 30 seconds, about 50$\times$ faster than the best existing methods. Extensive experiments demonstrate the efficiency and robustness of our method in segmenting various scenes, and its superior performance in downstream tasks such as object removal and inpainting. Demos and code will be available at https://github.com/florinshen/FlashSplat.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- Gaussian Grouping: Segment and Edit Anything in 3D Scenes2024
- Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis2024
- TensoRF: Tensorial Radiance Fields2022
- Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations2022
- Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation2022
- GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields2021
- NeX: Real-time View Synthesis with Neural Basis Expansion2021
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018