Simple and Effective Synthesis of Indoor 3D Scenes
Authors:
Open Access
- 26 June 2023
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (1) , 1169-1178
Abstract
We study the problem of synthesizing immersive 3D indoor scenes from one or a few images. Our aim is to generate high-resolution images and videos from novel viewpoints, including viewpoints that extrapolate far beyond the input images while maintaining 3D consistency. Existing approaches are highly complex, with many separately trained stages and components. We propose a simple alternative: an image-to-image GAN that maps directly from reprojections of incomplete point clouds to full high-resolution RGB-D images. On the Matterport3D and RealEstate10K datasets, our approach significantly outperforms prior work when evaluated by humans, as well as on FID scores. Further, we show that our model is useful for generative data augmentation. A vision-and-language navigation (VLN) agent trained with trajectories spatially-perturbed by our model improves success rate by up to 1.5% over a state of the art baseline on the mature R2R benchmark. Our code will be made available to facilitate generative data augmentation and applications to downstream robotics and embodied AI tasks.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- World-Consistent Video-to-Video Synthesis2020
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Point Encoder GAN: A deep learning model for 3D point cloud inpaintingNeurocomputing, 2020
- Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasksScientific Reports, 2019
- Image Inpainting for Irregular Holes Using Partial Convolutions2018
- MegaDepth: Learning Single-View Depth Prediction from Internet Photos2018
- Matterport3D: Learning from RGB-D Data in Indoor Environments2017
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- Deep Residual Learning for Image Recognition2016
- Novel view synthesis in tensor space2002