GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF
Authors:
Top Cited Papers
- 29 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1757-1763
Abstract
In this work, we tackle 6-DoF grasp detection for transparent and specular objects, which is an important yet challenging problem in vision-based robotic systems, due to the failure of depth cameras in sensing their geometry. We, for the first time, propose a multiview RGB-based 6-DoF grasp detection network, GraspNeRF, that leverages the generalizable neural radiance field (NeRF) to achieve material-agnostic object grasping in clutter. Compared to the existing NeRF-based 3-DoF grasp detection methods that rely on densely captured input images and time-consuming per-scene optimization, our system can perform zero-shot NeRF construction with sparse RGB inputs and reliably detect 6-DoF grasps, both in real-time. The proposed framework jointly learns generalizable NeRF and grasp detection in an end-to-end manner, optimizing the scene representation construction for the grasping. For training data, we generate a large-scale photorealistic domain-randomized synthetic dataset of grasping in cluttered tabletop scenes that enables direct transfer to the real world. Our extensive experiments in synthetic and real-world environments demonstrate that our method significantly outperforms all the baselines in all the experiments while remaining in real-time. Project page can be found at https://pku-epic.github.io/GraspNeRF.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping BaselineIEEE Robotics and Automation Letters, 2022
- Fuzzy-Depth Objects Grasping Based on FSG Algorithm and a Soft Robotic Hand2021
- GhostPose: Multi-view Pose Estimation of Transparent Objects for Robot Hand Grasping2021
- Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations2021
- RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images2021
- Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes2021
- GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping2020
- Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation2020
- Multi-Modal Transfer Learning for Grasping Transparent and Specular ObjectsIEEE Robotics and Automation Letters, 2020
- GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter2019