Imitating Tool-Based Garment Folding From a Single Visual Observation Using Hand-Object Graph Dynamics
- 1 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 20 (4) , 6245-6256
Abstract
Garment folding is a ubiquitous domestic task that is difficult to automate due to the highly deformable nature of fabrics. In this article, we propose a novel method of learning from demonstrations that enables robots to autonomously manipulate an assistive tool to fold garments. In contrast to traditional methods (that rely on low-level pixel features), our proposed solution uses a dense visual descriptor to encode the demonstration into a high-level hand-object graph (HoG) that allows to efficiently represent the interactions between the manipulated tool and robots. With that, we leverage graph neural network to autonomously learn the forward dynamics model from HoGs, then, given only a single demonstration, the imitation policy is optimized with a model predictive controller to accomplish the folding task. To validate the proposed approach, we conducted a detailed experimental study on a robotic platform instrumented with vision sensors and a custom-made end-effector that interacts with the folding board.Keywords
Funding Information
- Research Grants Council of Hong Kong (15212721, 15231023)
- Chinese National Engineering Research Centre for Steel Construction Hong Kong Branch (BBV8)
This publication has 29 references indexed in Scilit:
- Model-free reinforcement learning from expert demonstrations: a surveyArtificial Intelligence Review, 2021
- A survey of inverse reinforcement learning: Challenges, methods and progressArtificial Intelligence, 2021
- LaSeSOM: A Latent and Semantic Representation Framework for Soft Object ManipulationIEEE Robotics and Automation Letters, 2021
- Dynamic Cloth Manipulation with Deep Reinforcement Learning2020
- Self-Supervised Correspondence in Visuomotor Policy LearningIEEE Robotics and Automation Letters, 2019
- Zero-Shot Visual Imitation2018
- Fast Image Processing with Fully-Convolutional Networks2017
- Geodesic Convexity in Graphs2013
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programmingMathematical Programming, 2005