Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3403-3414
Abstract
In recent years, 3D representation learning has turned to 2D vision-language pre-trained models to overcome data scarcity challenges. However, existing methods simply transfer 2D alignment strategies, aligning 3D representations with single-view 2D images and coarse-grained parent category text. These approaches introduce information degradation and insufficient synergy issues, leading to performance loss. Information degradation arises from overlooking the fact that a 3D representation should be equivalent to a series of multi-view images and more fine-grained subcategory text. Insufficient synergy neglects the idea that a robust 3D representation should align with the joint vision-language space, rather than independently aligning with each modality. In this paper, we propose a multi-view joint modality modeling approach, termed JM3D, to obtain a unified representation for point cloud, text, and image. Specifically, a novel Structured Multimodal Organizer (SMO) is proposed to address the information degradation issue, which introduces contiguous multi-view images and hierarchical text to enrich the representation of vision and language modalities. A Joint Multi-modal Alignment (JMA) is designed to tackle the insufficient synergy problem, which models the joint modality by incorporating language knowledge into the visual modality. Extensive experiments on ModelNet40 and ScanObjectNN demonstrate the effectiveness of our proposed method, JM3D, which achieves state-of-the-art performance in zero-shot 3D classification. JM3D outperforms ULIP by approximately 4.3% on PointMLP and achieves an improvement of up to 6.5% accuracy on PointNet++ in top-1 accuracy for zero-shot 3D classification on ModelNet40. The source code and trained models for all our experiments are publicly available at https://github.com/Mr-Neko/JM3D.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (No.2022ZD0118201)
- The Natural Science Foundation of Fujian Province of China (No.2021J01002, No.2022J06001)
- The National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305, No. 62272401)
- The National Science Fund for Distinguished Young Scholars (No.62025603)
- China Postdoctoral Science Foundation (No.2023M732948)
This publication has 32 references indexed in Scilit:
- A Closer Look at Local Aggregation Operators in Point Cloud Analysis2020
- PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection2020
- DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing2019
- Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data2019
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers2019
- DGCNN: Disordered graph convolutional neural network based on the Gaussian mixture modelNeurocomputing, 2018
- 3D Semantic Parsing of Large-Scale Indoor Spaces2016
- VoxNet: A 3D Convolutional Neural Network for real-time object recognition2015
- The wave kernel signature: A quantum mechanical approach to shape analysis2011
- Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories2005