Unifying 3D Vision-Language Understanding via Promptable Queries
Authors:
- Baoxiong JiaBJBaoxiong Jia
Preprint
Abstract
A unified model for 3D vision-language (3D-VL) understanding is expected to take various scene representations and perform a wide range of tasks in a 3D scene. However, a considerable gap exists between existing methods and such a unified model, due to the independent application of representation and insufficient exploration of 3D multi-task training. In this paper, we introduce PQ3D, a unified model capable of using Promptable Queries to tackle a wide range of 3D-VL tasks, from low-level instance segmentation to high-level reasoning and planning. This is achieved through three key innovations: (1) unifying various 3D scene representations (i.e., voxels, point clouds, multi-view images) into a shared 3D coordinate space by segment-level grouping, (2) an attention-based query decoder for task-specific information retrieval guided by prompts, and (3) universal output heads for different tasks to support multi-task training. Tested across ten diverse 3D-VL datasets, PQ3D demonstrates impressive performance on these tasks, setting new records on most benchmarks. Particularly, PQ3D improves the state-of-the-art on ScanNet200 by 1.8% (AP), ScanRefer by 5.4% (acc@0.5), Multi3DRefer by 11.7% (F1@0.5), and Scan2Cap by 13.4% (CIDEr@0.5). Moreover, PQ3D supports flexible inference with individual or combined forms of available 3D representations, e.g., solely voxel input.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding2024
- Language-Grounded Indoor 3D Semantic Segmentation in the Wild2022
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds2022
- Scan2Cap: Context-aware Dense Captioning in RGB-D Scans2021
- ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes2020
- Interactive Image Segmentation With First Click Attention2020
- PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation2020
- 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks2019
- Efficient Graph-Based Image SegmentationInternational Journal of Computer Vision, 2004