Learning to Grasp in Clutter with Interactive Visual Failure Prediction
Authors:
- 13 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 18172-18178
Abstract
Modern warehouses process millions of unique objects which are often stored in densely packed containers. To automate tasks in this environment, a robot must be able to pick diverse objects from highly cluttered scenes. Real-world learning is a promising approach, but executing picks in the real world is time-consuming, can induce costly failures, and often requires extensive human intervention, which causes operational burden and limits the scope of data collection and deployments. In this work, we leverage interactive probes to visually evaluate grasps in clutter without fully executing picks, a capability we refer to as Interactive Visual Failure Prediction (IVFP). This enables autonomous verification of grasps during execution to avoid costly downstream failures as well as autonomous reward assignment, providing supervision to continuously shape and improve grasping behavior as the robot gathers experience in the real world, without constantly requiring human intervention. Through experiments on a Stretch RE1 robot, we study the effect that IVFP has on performance - both in terms of effective data throughput and success rate, and show that this approach leads to grasping policies that outperform policies trained with human supervision alone, while requiring significantly less human intervention. Code, datasets, and videos available at https://robo-ivfp.github.ioKeywords
This publication has 18 references indexed in Scilit:
- The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human EnvironmentsIEEE International Conference on Robotics and Automation (ICRA), 2022
- Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes2021
- A Survey on Learning-Based Robotic GraspingCurrent Robotics Reports, 2020
- Unified Perceptual Parsing for Scene Understanding2018
- The “Something Something” Video Database for Learning and Evaluating Visual Common Sense2017
- Feature Pyramid Networks for Object Detection2017
- Learning to Switch Between Sensorimotor Primitives Using Multimodal Haptic Signals2016
- Leveraging big data for grasp planning2015
- Skill learning and task outcome prediction for manipulation2011
- Failure detection in assembly: Force signature analysis2010