Real-time Large-scale Deformation of Gaussian Splatting
Open Access
- 19 November 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 43 (6) , 1-17
Abstract
Neural implicit representations, including Neural Distance Fields and Neural Radiance Fields, have demonstrated significant capabilities for reconstructing surfaces with complicated geometry and topology, and generating novel views of a scene. Nevertheless, it is challenging for users to directly deform or manipulate these implicit representations with large deformations in a real-time fashion. Gaussian Splatting (GS) has recently become a promising method with explicit geometry for representing static scenes and facilitating high-quality and real-time synthesis of novel views. However, it cannot be easily deformed due to the use of discrete Gaussians and the lack of explicit topology. To address this, we develop a novel GS-based method (GaussianMesh) that enables interactive deformation. Our key idea is to design an innovative mesh-based GS representation, which is integrated into Gaussian learning and manipulation. 3D Gaussians are defined over an explicit mesh, and they are bound with each other: the rendering of 3D Gaussians guides the mesh face split for adaptive refinement, and the mesh face split directs the splitting of 3D Gaussians. Moreover, the explicit mesh constraints help regularize the Gaussian distribution, suppressing poor-quality Gaussians ( e.g. , misaligned Gaussians, long-narrow shaped Gaussians), thus enhancing visual quality and reducing artifacts during deformation. Based on this representation, we further introduce a large-scale Gaussian deformation technique to enable deformable GS, which alters the parameters of 3D Gaussians according to the manipulation of the associated mesh. Our method benefits from existing mesh deformation datasets for more realistic data-driven Gaussian deformation. Extensive experiments show that our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate (65 FPS on average on a single commodity GPU).Keywords
Funding Information
- Innovation Funding of ICT, CAS (E461020)
- National Natural Science Foundation of China (62322210, 62302484)
- Beijing Municipal Natural Science Foundation for Distinguished Young Scholars (JQ21013)
- Beijing Municipal Science and Technology Commission, Adminitrative Commission of Zhongguancun Science Park (Z231100005923031)
This publication has 36 references indexed in Scilit:
- Recent advances in implicit representation-based 3D shape generationVisual Intelligence, 2024
- Neural Cages for Detail-Preserving 3D Deformations2020
- Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion2020
- BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks2020
- Proxy-driven free-form deformation by topology-adjustable control latticeComputers & Graphics, 2020
- Learning Implicit Fields for Generative Shape Modeling2019
- Occupancy Networks: Learning 3D Reconstruction in Function Space2019
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images2018
- A Papier-Mache Approach to Learning 3D Surface Generation2018
- The role of prior in image based 3D modeling: a surveyFrontiers of Computer Science, 2016