2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds
Authors:
- Shenghui CuiSCShenghui Cui
Preprint
Abstract
As camera and LiDAR sensors capture complementary information used in autonomous driving, great efforts have been made to develop semantic segmentation algorithms through multi-modality data fusion. However, fusion-based approaches require paired data, i.e., LiDAR point clouds and camera images with strict point-to-pixel mappings, as the inputs in both training and inference, which seriously hinders their application in practical scenarios. Thus, in this work, we propose the 2D Priors Assisted Semantic Segmentation (2DPASS), a general training scheme, to boost the representation learning on point clouds, by fully taking advantage of 2D images with rich appearance. In practice, by leveraging an auxiliary modal fusion and multi-scale fusion-to-single knowledge distillation (MSFSKD), 2DPASS acquires richer semantic and structural information from the multi-modal data, which are then online distilled to the pure 3D network. As a result, equipped with 2DPASS, our baseline shows significant improvement with only point cloud inputs. Specifically, it achieves the state-of-the-arts on two large-scale benchmarks (i.e. SemanticKITTI and NuScenes), including top-1 results in both single and multiple scan(s) competitions of SemanticKITTI.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds2022
- 3D-to-2D Distillation for Indoor Scene Parsing2021
- LatticeNet: Fast Point Cloud Segmentation Using Permutohedral Lattices2020
- 3D-MiniNet: Learning a 2D Representation From Point Clouds for Fast and Efficient 3D LIDAR Semantic SegmentationIEEE Robotics and Automation Letters, 2020
- nuScenes: A Multimodal Dataset for Autonomous Driving2020
- Dynamic Graph CNN for Learning on Point CloudsACM Transactions on Graphics, 2019
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017