Multi-Stage Reinforcement Learning for Non-Prehensile Manipulation
- 11 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (7) , 6712-6719
Abstract
Manipulating objects without grasping them facilitates complex tasks, known as non-prehensile manipulation. Most previous methods are limited to learning a single skill to manipulate objects with primitive shapes and are unserviceable for flexible object manipulation that requires a combination of multiple skills. We explore skill-unconstrained non-prehensile manipulation and propose a Multi-stage Reinforcement Learning for Non-prehensile Manipulation ( MRLNM ), which calculates an intermediate state between the initial and goal states and divides the task into multiple stages for sequential learning. At each stage, the policy takes the desired 6-DOF object pose as the goal, and proposes a spatially-continuous action, allowing the robot to explore arbitrary skills to accomplish the task. To handle objects with different shapes, we propose a State-Goal Fusion Representation (SGF-Representation) to represent observations and goals as point clouds with motion, which improves the policy's perception of scene layout and task goal. To improve sample efficiency, we propose a Spatially-Reachable Distance Metric (SR-Distance) to approximately measure the shortest distance between two points without intersecting the scene. We evaluate MRLNM on an occluded grasping task which aims to grasp the object in initially occluded configurations. MRLNM demonstrates generalization to unseen objects with shapes outside the training distribution and can be transferred to the real world with zero-shot transfer, achieving a 95% success rate.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (U22A2058, 62176138, 62176136)
- National Key R&D Program of China (2018YFB1305300)
- shandong provincial key research and development program (2019JZZY010130, 2020CXGC010207)
This publication has 24 references indexed in Scilit:
- Demonstration-guided Optimal Control for Long-term Non-prehensile Planar Manipulation2023
- Contact Mode Guided Motion Planning for Quasidynamic Dexterous Manipulation in 3D2022
- Fast High-Quality Tabletop Rearrangement in Bounded Workspace2022
- OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation2021
- Learning Visual Affordances with Target-Orientated Deep Q-Network to Grasp Objects by Harnessing Environmental Fixtures2021
- Learning Pregrasp Manipulation of Objects from Ungraspable Poses2020
- MuJoCo: A physics engine for model-based control2012
- Close-range scene segmentation and reconstruction of 3D point cloud maps for mobile manipulation in domestic environments2009
- A visibility-based accessibility analysis of the grasp points for real-time manipulation2005
- A method for registration of 3-D shapesIEEE Transactions on Pattern Analysis and Machine Intelligence, 1992