Learning 6-DoF Fine-Grained Grasp Detection Based on Part Affordance Grounding
- 2 May 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 22 (15455955) , 15200-15214
Abstract
Robotic grasping is a fundamental ability for a robot to interact with the environment. Current methods focus on how to obtain a stable and reliable grasping pose in object level, while little work has been studied on part (shape)-wise grasping which is related to fine-grained grasping and robotic affordance. Parts can be seen as atomic elements to compose an object, which contains rich semantic knowledge and a strong correlation with affordance. However, lacking a large part-wise 3D robotic dataset limits the development of part representation learning and downstream applications. In this paper, we propose a new large Language-guided SHape grAsPing datasEt (named LangSHAPE) to promote 3D part-level affordance and grasping ability learning. From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD), including a novel 3D part language grounding model and a partaware grasp pose detection model, in which explicit language input from human or large language models (LLMs) could guide a robot to generate part-level 6-DoF grasping pose with textual explanation. Our method combines the advantages of humanrobot collaboration and LLMs’ planning ability using explicit language as a symbolic intermediate. To evaluate the effectiveness of our proposed method, we perform 3D part grounding and fine-grained grasp detection experiments on both simulation and physical robot settings, following language instructions across different degrees of textual complexity. Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks. Our dataset and code are available on our project website https://sites.google.com/view/lang-shape.Keywords
All Related Versions
Funding Information
- Postdoctoral Fellowship Program of CPSF (GZC20232292)
- National Natural Science Foundation of China (T2125009)
- National Natural Science Foundation of China (92048302)
- “Pioneer” Research and Development Program of Zhejiang (2023C03007)
- Suzhou Key Laboratory of Artificial Intelligence and Social Governance Technologies (SZS2023007)
- Smart Social Governance Technology and Innovative Application Platform (YZCXPT2023101)
This publication has 69 references indexed in Scilit:
- Deep Robotic Grasping Prediction with Hierarchical RGB-D FusionInternational Journal of Control, Automation, and Systems, 2022
- Composing Pick-and-Place Tasks by Grounding Language2021
- ScanRefer: 3D Object Localization in RGB-D Scans Using Natural Language2020
- GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping2020
- Grasp Pose Detection with Affordance-based Task Constraint Learning in Single-view Point CloudsJournal of Intelligent & Robotic Systems, 2020
- Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common SenseEngineering, 2020
- PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding2019
- Weakly Supervised Affordance Detection2017
- WordNet2010
- Learning grasping points with shape contextRobotics and Autonomous Systems, 2010