PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models
Authors:
- Minghua LiuMLMinghua Liu
- Yinhao ZhuYZYinhao Zhu
- Hong CaiHCHong Cai
- Shizhong HanSHShizhong Han
- Zhan LingZLZhan Ling
- Fatih PorikliFPFatih Porikli
- Hao SuHSHao Su
Preprint
Abstract
Generalizable 3D part segmentation is important but challenging in vision and robotics. Training deep models via conventional supervised methods requires large-scale 3D datasets with fine-grained part annotations, which are costly to collect. This paper explores an alternative way for low-shot part segmentation of 3D point clouds by leveraging a pretrained image-language model, GLIP, which achieves superior performance on open-vocabulary 2D detection. We transfer the rich knowledge from 2D to 3D through GLIP-based part detection on point cloud rendering and a novel 2D-to-3D label lifting algorithm. We also utilize multi-view 3D priors and few-shot prompt tuning to boost performance significantly. Extensive evaluation on PartNet and PartNet-Mobility datasets shows that our method enables excellent zero-shot 3D part segmentation. Our few-shot version not only outperforms existing few-shot approaches by a large margin but also achieves highly competitive results compared to the fully supervised counterpart. Furthermore, we demonstrate that our method can be directly applied to iPhone-scanned point clouds without significant domain gaps.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Semi-supervised 3D shape segmentation with multilevel consistency and part substitutionComputational Visual Media, 2023
- MvDeCor: Multi-view Dense Correspondence Learning for Fine-Grained 3D Segmentation2022
- Language-Grounded Indoor 3D Semantic Segmentation in the Wild2022
- Approximate convex decomposition for 3D meshes with collision-aware concavity and tree searchACM Transactions on Graphics, 2022
- Voxel-informed Language Grounding2022
- ICM-3D: Instantiated Category Modeling for 3D Instance SegmentationIEEE Robotics and Automation Letters, 2021
- Self-Supervised Learning of Local Features in 3D Point Clouds2020
- Dynamic Graph CNN for Learning on Point CloudsACM Transactions on Graphics, 2019
- Cut Pursuit: Fast Algorithms to Learn Piecewise Constant Functions on General Weighted GraphsSIAM Journal on Imaging Sciences, 2017
- A scalable active framework for region annotation in 3D shape collectionsACM Transactions on Graphics, 2016