Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks
Authors:
- 1 November 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1010-1017
Abstract
Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or reward model, few efforts focused on the choice of action space (e.g. joint or end-effector space, position, velocity, etc.). However, studies in robot motion control indicate that choosing an action space that conforms to the characteristics of the task can simplify exploration and improve robustness to disturbances. This paper studies the effect of different action spaces in deep RL and advocates for variable impedance control in end-effector space (VICES) as an advantageous action space for constrained and contact-rich tasks. We evaluate multiple action spaces on three prototypical manipulation tasks: Path Following (task with no contact), Door Opening (task with kinematic constraints), and Surface Wiping (task with continuous contact). We show that VICES improves sample efficiency, maintains low energy consumption, and ensures safety across all three experimental setups. Further, RL policies learned with VICES can transfer across different robot models in simulation, and from simulation to real for the same robot. Further information is available at https://stanfordvl.github.io/vices.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Dynamical Movement Primitives: Learning Attractor Models for Motor BehaviorsNeural Computation, 2013
- Optimal distribution of contact forces with inverse-dynamics controlThe International Journal of Robotics Research, 2013
- Learned graphical models for probabilistic planning provide a new class of movement primitivesFrontiers in Computational Neuroscience, 2013
- Learning variable impedance controlThe International Journal of Robotics Research, 2011
- Learning impedance control of antagonistic systems based on stochastic optimization principlesThe International Journal of Robotics Research, 2010
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- Computational approaches to motor learning by imitationPhilosophical Transactions of the Royal Society B: Biological Sciences, 2003
- Specification of force-controlled actions in the "task frame formalism"-a synthesisIEEE Transactions on Robotics and Automation, 1996
- Inertial Properties in Robotic Manipulation: An Object-Level FrameworkThe International Journal of Robotics Research, 1995
- Impedance Control: An Approach to Manipulation: Part II—ImplementationJournal of Dynamic Systems, Measurement, and Control, 1985