NeuralLift-360: Lifting an in-the-Wild 2D Photo to A 3D Object with 360° Views
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4479-4489
Abstract
Virtual reality and augmented reality (XR) bring increasing demand for 3D content generation. However, creating high-quality 3D content requires tedious work from a human expert. In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360° views that corresponds well with the given reference image. By conditioning on the reference image, our model can fulfill the everlasting curiosity for synthesizing novel views of objects from images. Our technique sheds light on a promising direction of easing the workflows for 3D artists and XR designers. We propose a novel framework, dubbed NeuralLift-360, that utilizes a depth-aware neural radiance representation (NeRF) and learns to craft the scene guided by denoising diffusion models. By introducing a ranking loss, our NeuralLift-360 can be guided with rough depth estimation in the wild. We also adopt a CLIP-guided sampling strategy for the diffusion prior to provide coherent guidance. Extensive experiments demonstrate that our NeuralLift-360 significantly outperforms existing state-of-the-art baselines. Project page: https://vita-group.github.io/NeuralLift-360/Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization2019
- Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes2019
- MegaDepth: Learning Single-View Depth Prediction from Internet Photos2018
- Learning Depth from Monocular Videos Using Direct Methods2018
- Ordinal Depth Supervision for 3D Human Pose Estimation2018
- Scaling CNNs for High Resolution Volumetric Reconstruction from a Single Image2017
- A Point Set Generation Network for 3D Object Reconstruction from a Single Image2017
- Nonlinear disparity mapping for stereoscopic 3DACM Transactions on Graphics, 2010
- Numerical methods for shape-from-shading: A new survey with benchmarksComputer Vision and Image Understanding, 2008
- A geometric approach to shape from defocusIEEE Transactions on Pattern Analysis and Machine Intelligence, 2005