RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models
Authors:
Preprint
Abstract
We address the challenge of recovering an underlying scene geometry and colors from a sparse set of RGBD view observations. In this work, we present a new solution termed RGBD$^2$ that sequentially generates novel RGBD views along a camera trajectory, and the scene geometry is simply the fusion result of these views. More specifically, we maintain an intermediate surface mesh used for rendering new RGBD views, which subsequently becomes complete by an inpainting network; each rendered RGBD view is later back-projected as a partial surface and is supplemented into the intermediate mesh. The use of intermediate mesh and camera projection helps solve the tough problem of multi-view inconsistency. We practically implement the RGBD inpainting network as a versatile RGBD diffusion model, which is previously used for 2D generative modeling; we make a modification to its reverse diffusion process to enable our use. We evaluate our approach on the task of 3D scene synthesis from sparse RGBD inputs; extensive experiments on the ScanNet dataset demonstrate the superiority of our approach over existing ones. Project page: https://jblei.site/proj/rgbd-diffusion.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion ModelsACM Transactions on Graphics, 2023
- NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior2023
- Weakly Supervised 3D Point Cloud Segmentation via Multi-Prototype LearningIEEE Transactions on Circuits and Systems for Video Technology, 2023
- Image Super-Resolution Via Iterative RefinementIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Instant neural graphics primitives with a multiresolution hash encodingACM Transactions on Graphics, 2022
- SRDiff: Single image super-resolution with diffusion probabilistic modelsNeurocomputing, 2022
- Learning and Meshing From Deep Implicit Surface Networks Using an Efficient Implementation of Analytic MarchingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- SkeletonNet: A Topology-Preserving Solution for Learning Mesh Reconstruction of Object Surfaces From RGB ImagesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Deep Generative Modeling for Scene Synthesis via Hybrid RepresentationsACM Transactions on Graphics, 2020
- Deep convolutional priors for indoor scene synthesisACM Transactions on Graphics, 2018