LASO: Language-Guided Affordance Segmentation on 3D Object
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14251-14260
Abstract
Segmenting affordance in 3D data is key for bridging perception and action in robots. Existing efforts mostly focus on the visual side and overlook the affordance knowledge from a semantic aspect. This oversight not only limits their generalization to unseen objects, but more importantly, hinders their synergy with large language models (LLMs) which are excellent task planners that can decompose an overarching command into agent-actionable instructions. With this regard, we propose a novel task, Language-guided Affordance Segmentation on 3D Object (LASO), which challenges a model to segment a 3D object's part relevant to a given affordance question. To facilitate the task, we contribute a dataset comprising 19,751 point-question pairs, covering 8434 object shapes and 870 expert-crafted questions. As a pioneer solution, we further propose PointRefer, which highlights an adaptive fusion module to identify target affordance regions at different scales. To ensure a text-aware segmentation, we adopt a set of affordance queries conditioned on linguistic cues to generate dynamic kernels. These kernels are further used to convolute with point features and generate a segmentation mask. Comprehensive experiments and analyses validate PointRefer's effectiveness. With these efforts, We hope that LASO can steer the direction of 3D affordance, guiding it towards enhanced integration with the evolving capabilities of LLMs. Code and data are available at https://github.com/yl3800/LASO.Keywords
Funding Information
- National Natural Science Foundation of China (92270114)
- Agency for Science, Technology and Research (A*STAR) (M23L7b0021)
This publication has 29 references indexed in Scilit:
- Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds2022
- Invariant Grounding for Video Question Answering2022
- Learning Affordance Grounding from Exocentric Images2022
- 3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding2021
- ScanRefer: 3D Object Localization in RGB-D Scans Using Natural Language2020
- ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes2020
- End-to-End Object Detection with Transformers2020
- Object affordance detection with relationship-aware networkNeural Computing and Applications, 2019
- Towards affordance detection for robot manipulation using affordance for parts and parts for affordanceAutonomous Robots, 2018
- Demo2Vec: Reasoning Object Affordances from Online Videos2018