SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 19946-19956
Abstract
3D occupancy prediction is an important task for the robustness of vision-centric autonomous driving, which aims to predict whether each point is occupied in the surrounding 3D space. Existing methods usually require 3D occupancy labels to produce meaningful results. However, it is very laborious to annotate the occupancy status of each voxel. In this paper, we propose SelfOcc to explore a self-supervised way to learn 3D occupancy using only video sequences. We first transform the images into the 3D space (e.g., bird's eye view) to obtain 3D representation of the scene. We directly impose constraints on the 3D representations by treating them as signed distance fields. We can then render 2D images of previous and future frames as self-supervision signals to learn the 3D representations. We propose an MVS-embedded strategy to directly optimize the SDF-induced weights with multiple depth proposals. Our SelfOcc out-performs the previous best method SceneRF by 58.7% using a single frame as input on SemanticKITTI and is the first self-supervised work that produces reasonable 3D occupancy for surround cameras on nuScenes. SelfOcc produces high-quality depth and achieves state-of-the-art results on novel depth synthesis, monocular depth estimation, and surround-view depth estimation on the SemanticKITTI, KITTI-2015, and nuScenes, respectively. Code: https://github.com/huang-yh/SelfOcc.Keywords
All Related Versions
This publication has 65 references indexed in Scilit:
- MPPNet: Multi-frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection2022
- TensoRF: Tensorial Radiance Fields2022
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- DevNet: Self-supervised Monocular Depth Learning via Density Volume Construction2022
- Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D2020
- Feature-Metric Loss for Self-supervised Learning of Depth and Egomotion2020
- SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion2020
- LMSCNet: Lightweight Multiscale 3D Semantic Completion2020
- Efficient Semantic Scene Completion Network with Spatial Group Convolution2018
- Optical models for direct volume renderingIEEE Transactions on Visualization and Computer Graphics, 1995