Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention
- 15 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (2) , 760-767
Abstract
Sequential multi-step cloth manipulation is a challenging problem in robotic manipulation, requiring a robot to perceive the cloth state and plan a sequence of chained actions leading to the desired state. Most previous works address this problem in a goal-conditioned way, and goal observation must be given for each specific task and cloth configuration, which is not practical and efficient. Thus, we present a novel multi-step cloth manipulation planning framework named Foldformer. Foldformer can complete similar tasks with only a general demonstration and utilize a space-time attention mechanism to capture the instruction information behind this demonstration. We experimentally evaluate Foldsformer on four representative sequential multi-step manipulation tasks and show that Foldsformer significantly outperforms state-of-the-art approaches in simulation. Foldformer can complete multi-step cloth manipulation tasks even when configurations of the cloth (e.g., size and pose) vary from configurations in the general demonstrations. Furthermore, our approach can be transferred from simulation to the real world without additional training or domain randomization. Despite training on rectangular clothes, we also show that our approach can generalize to unseen cloth shapes (T-shirts and shorts).Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2022YFB4701400/4701402)
- National Natural Science Foundation of China (U1813216, U21B6002, 62203260)
- Shenzhen Science Fund for Distinguished Young Scholars (RCJC20210706091946001)
- Guangdong Young Talent with Scientific and Technological Innovation (2019TQ05Z111)
- China Postdoctoral Science Foundation (2022M711823)
This publication has 21 references indexed in Scilit:
- Zero-Shot Visual Imitation2018
- Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning2017
- Combining self-supervised learning and imitation for vision-based rope manipulation2017
- Folding Clothes Autonomously: A Complete PipelineIEEE Transactions on Robotics, 2016
- MoveIt! [ROS Topics]IEEE Robotics & Automation Magazine, 2012
- Soft Material Modeling for Robotic ManipulationApplied Mechanics and Materials, 2012
- A geometric approach to robotic laundry foldingThe International Journal of Robotics Research, 2011
- Motion planning for dynamic folding of a cloth with two high-speed robot hands and two high-speed sliders2011
- Vision-guided robotic fabric manipulation for apparel manufacturingIEEE Control Systems Magazine, 1988
- A Computational Approach to Edge DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 1986