PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 21736-21746
Abstract
Generalizable 3D part segmentation is important but challenging in vision and robotics. Training deep models via conventional supervised methods requires large-scale 3D datasets with fine-grained part annotations, which are costly to collect. This paper explores an alternative way for low-shot part segmentation of 3D point clouds by leveraging a pretrained image-language model, GLIP. which achieves superior performance on open-vocabulary 2D detection. We transfer the rich knowledge from 2D to 3D through GLIP-based part detection on point cloud rendering and a novel 2D-to-3D label lifting algorithm. We also utilize multi-view 3D priors and few-shot prompt tuning to boost performance significantly. Extensive evaluation on PartNet and PartNet-Mobility datasets shows that our method enables excellent zero-shot 3D part segmentation. Our few-shot version not only outperforms existing few-shot approaches by a large margin but also achieves highly competitive results compared to the fully supervised counterpart. Furthermore, we demonstrate that our method can be directly applied to iPhone-scanned point clouds without significant domain gaps.Keywords
All Related Versions
This publication has 52 references indexed in Scilit:
- StructureNetACM Transactions on Graphics, 2019
- Dynamic Graph CNN for Learning on Point CloudsACM Transactions on Graphics, 2019
- Associatively Segmenting Instances and Semantics in Point Clouds2019
- 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans2019
- Relation-Shape Convolutional Neural Network for Point Cloud Analysis2019
- SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation2018
- Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs2018
- Cut Pursuit: Fast Algorithms to Learn Piecewise Constant Functions on General Weighted GraphsSIAM Journal on Imaging Sciences, 2017
- A scalable active framework for region annotation in 3D shape collectionsACM Transactions on Graphics, 2016
- A 3D shape segmentation approach for robot grasping by partsRobotics and Autonomous Systems, 2012