GPT4Point: A Unified Framework for Point-Language Understanding and Generation
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 26407-26417
Abstract
Multimodal Large Language Models (MLLMs) have excelled in 2D image-text comprehension and image generation. Still, their understanding of the 3D world needs to be improved, limiting progress in 3D language understanding and generation. To solve this problem, we introduce GPT4Point, an innovative, groundbreaking point-language multimodal model explicitly designed for unified 3D object understanding and generation within the MLLMframework. GPT4Point, as a powerful 3D MLLM, can seamlessly execute point-text reference tasks such as point-cloud captioning and Q&A. Additionally, GPT4Point is equipped with advanced capabilities for controllable 3D generation, and it can get high-quality results through a low-quality point-text feature that maintains geometric shapes and colors. We develop Pyramid-XL, a point-language dataset annotation engine, to support the expansive needs of 3D object-text pairs. It constructs a large-scale database of over 1M objects of varied text granularity levels from the Objaverse-XL dataset, essential for training GPT4Point. A comprehensive benchmark has been proposed to evaluate 3D point-language understanding capabilities. In extensive evaluations, GPT4Point has demonstrated superior performance in understanding and generation.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62201484)
This publication has 20 references indexed in Scilit:
- OCBEV: Object-Centric BEV Transformer for Multi-View 3D Object Detection2024
- Magic3D: High-Resolution Text-to-3D Content Creation2023
- ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding2023
- OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation2023
- Masked Autoencoders for Point Cloud Self-supervised Learning2022
- PointCLIP: Point Cloud Understanding by CLIP2022
- Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts2021
- LVIS: A Dataset for Large Vocabulary Instance Segmentation2019