Learning visual-based deformable object rearrangement with local graph neural networks
Open Access
- 12 April 2023
- journal article
- research article
- Published by Springer Nature in Complex & Intelligent Systems
- Vol. 9 (5) , 5923-5936
Abstract
Goal-conditioned rearrangement of deformable objects (e.g. straightening a rope and folding a cloth) is one of the most common deformable manipulation tasks, where the robot needs to rearrange a deformable object into a prescribed goal configuration with only visual observations. These tasks are typically confronted with two main challenges: the high dimensionality of deformable configuration space and the underlying complexity, nonlinearity and uncertainty inherent in deformable dynamics. To address these challenges, we propose a novel representation strategy that can efficiently model the deformable object states with a set of keypoints and their interactions. We further propose local-graph neural network (GNN), a light local GNN learning to jointly model the deformable rearrangement dynamics and infer the optimal manipulation actions (e.g. pick and place) by constructing and updating two dynamic graphs. Both simulated and real experiments have been conducted to demonstrate that the proposed dynamic graph representation shows superior expressiveness in modeling deformable rearrangement dynamics. Our method reaches much higher success rates on a variety of deformable rearrangement tasks (96.3% on average) than state-of-the-art method in simulation experiments. Besides, our method is much more lighter and has a 60% shorter inference time than state-of-the-art methods. We also demonstrate that our method performs well in the multi-task learning scenario and can be transferred to real-world applications with an average success rate of 95% by solely fine tuning a keypoint detector. A supplementary video can be found at https://youtu.be/AhwTQo6fCM0.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- GraspVDN: scene-oriented grasp estimation by learning vector representations of graspsComplex & Intelligent Systems, 2021
- Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks2021
- Modeling, learning, perception, and control methods for deformable object manipulationScience Robotics, 2021
- Dynamic Manipulation of Deformable Objects With Implicit IntegrationIEEE Robotics and Automation Letters, 2021
- Learning to Manipulate Deformable Objects without Demonstrations2020
- Deep Reinforcement Learning for Robotic Pushing and Picking in Cluttered Environment2019
- Track deformable objects from point clouds with structure preserved registrationThe International Journal of Robotics Research, 2019
- State estimation for deformable objects by point registration and dynamic simulation2017
- Combining self-supervised learning and imitation for vision-based rope manipulation2017
- A geometric approach to robotic laundry foldingThe International Journal of Robotics Research, 2011