MonoScene: Monocular 3D Semantic Scene Completion
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3981-3991
Abstract
MonoScene proposes a 3D Semantic Scene Completion (SSC) framework, where the dense geometry and semantics of a scene are inferred from a single monocular RGB image. Different from the SSC literature, relying on 2.5 or 3D input, we solve the complex problem of 2D to 3D scene reconstruction while jointly inferring its semantics. Our framework relies on successive 2D and 3D UNets, bridged by a novel 2D-3D features projection inspired by optics, and introduces a 3D context relation prior to enforce spatio-semantic consistency. Along with architectural contributions, we introduce novel global scene and local frustums losses. Experiments show we outperform the literature on all metries and datasets while hallucinating plausible scenery even beyond the camera field of view. Our code and trained models are available at https://github.com/cv-rits/MonoScene.Keywords
All Related Versions
This publication has 56 references indexed in Scilit:
- High-Precision Depth Estimation with the 3D LiDAR and Stereo Fusion2018
- RoomNet: End-to-End Room Layout Estimation2017
- IM2CAD2017
- 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions2017
- Semantic Scene Completion from a Single Depth Image2017
- A Point Set Generation Network for 3D Object Reconstruction from a Single Image2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- Geometric context from a single image2005
- Shape-from-shading: a surveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 1999