Segment, Lift and Fit: Automatic 3D Shape Labeling from 2D Prompts
Preprint
Abstract
This paper proposes an algorithm for automatically labeling 3D objects from 2D point or box prompts, especially focusing on applications in autonomous driving. Unlike previous arts, our auto-labeler predicts 3D shapes instead of bounding boxes and does not require training on a specific dataset. We propose a Segment, Lift, and Fit (SLF) paradigm to achieve this goal. Firstly, we segment high-quality instance masks from the prompts using the Segment Anything Model (SAM) and transform the remaining problem into predicting 3D shapes from given 2D masks. Due to the ill-posed nature of this problem, it presents a significant challenge as multiple 3D shapes can project into an identical mask. To tackle this issue, we then lift 2D masks to 3D forms and employ gradient descent to adjust their poses and shapes until the projections fit the masks and the surfaces conform to surrounding LiDAR points. Notably, since we do not train on a specific dataset, the SLF auto-labeler does not overfit to biased annotation patterns in the training set as other methods do. Thus, the generalization ability across different datasets improves. Experimental results on the KITTI dataset demonstrate that the SLF auto-labeler produces high-quality bounding box annotations, achieving an AP@0.5 IoU of nearly 90\%. Detectors trained with the generated pseudo-labels perform nearly as well as those trained with actual ground-truth annotations. Furthermore, the SLF auto-labeler shows promising results in detailed shape predictions, providing a potential alternative for the occupancy annotation of dynamic objects.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection2022
- Multimodal Transformer for Automatic 3D Annotation and Object Detection2022
- Weakly Supervised 3D Object Detection from Lidar Point Cloud2020
- DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation2019
- Hybrid Task Cascade for Instance Segmentation2019
- 3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare2018
- Simultaneous Monocular 2D Segmentation, 3D Pose Recovery and 3D Reconstruction2013
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012
- Signed Distance Functions2003
- Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography1987