ProtoTransfer: Cross-Modal Prototype Transfer for Point Cloud Segmentation
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3314-3324
Abstract
Knowledge transfer from multi-modal, i.e., LiDAR points and images, to a single LiDAR modal can take advantage of complimentary information from modal-fusion but keep a single modal inference speed, showing a promising direction for point cloud semantic segmentation in autonomous driving. Recent advances in point cloud segmentation distill knowledge from strictly aligned point-pixel fusion features while leaving a large number of unmatched image pixels unexplored and unmatched LiDAR points under-benefited. In this paper, we propose a novel approach, named ProtoTransfer, which not only fully exploits image representations but also transfers the learned multi-modal knowledge to all point cloud features. Specifically, based on the basic multi-modal learning framework, we build up a class-wise prototype bank from the strictly-aligned fusion features and encourage all the point cloud features to learn from the prototypes during model training. Moreover, to exploit the massive unmatched point and pixel features, we use a pseudo-labeling scheme and further accumulate these features into the class-wise prototype bank with a carefully designed fusion strategy. Without bells and whistles, our approach demonstrates superior performance over the published state-of-the-arts on two large-scale benchmarks, i.e., nuScenes and SemanticKITTI, and ranks 2nd on the competitive nuScenes Lidarseg challenge leaderboard.Keywords
Funding Information
- Fundamental Research Funds for the Central Universities
This publication has 39 references indexed in Scilit:
- 2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds2022
- Learning 3D Semantic Segmentation with only 2D Image Supervision2021
- Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution2020
- SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation2020
- Unsupervised Feature Learning via Non-parametric Instance Discrimination2018
- Tangent Convolutions for Dense Prediction in 3D2018
- Deep Mutual Learning2018
- Deep Residual Learning for Image Recognition2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Fully convolutional networks for semantic segmentation2015