CLIP^2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
Authors:
- Xiaodan LiangXLXiaodan Liang
- Hang XuHXHang Xu
Preprint
Abstract
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks. However, due to the limited Text-3D data pairs, adapting the success of 2D Vision-Language Models (VLM) to the 3D space remains an open problem. Existing works that leverage VLM for 3D understanding generally resort to constructing intermediate 2D representations for the 3D data, but at the cost of losing 3D geometry information. To take a step toward open-world 3D vision understanding, we propose Contrastive Language-Image-Point Cloud Pretraining (CLIP^2) to directly learn the transferable 3D point cloud representation in realistic scenarios with a novel proxy alignment mechanism. Specifically, we exploit naturally-existed correspondences in 2D and 3D scenarios, and build well-aligned and instance-based text-image-point proxies from those complex scenarios. On top of that, we propose a cross-modal contrastive objective to learn semantic and instance-level aligned point cloud representation. Experimental results on both indoor and outdoor scenarios show that our learned 3D representation has great transfer ability in downstream tasks, including zero-shot and few-shot 3D recognition, which boosts the state-of-the-art methods by large margins. Furthermore, we provide analyses of the capability of different representations in real scenarios and present the optional ensemble scheme.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Masked Autoencoders for Point Cloud Self-supervised Learning2022
- Masked Discrimination for Self-supervised Learning on Point Clouds2022
- PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding2020
- H3DNet: 3D Object Detection Using Hybrid Geometric Primitives2020
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset2020
- SARPNET: Shape attention regional proposal network for liDAR-based 3D object detectionNeurocomputing, 2020
- VoteNet: A Deep Learning Label Fusion Method for Multi-atlas Segmentation2019
- PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud2019
- LVIS: A Dataset for Large Vocabulary Instance Segmentation2019
- DGCNN: A convolutional neural network over large-scale labeled graphsNeural Networks, 2018