Trajectory Optimization for Unknown Constrained Systems using Reinforcement Learning
Open Access
- 1 November 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3487-3494
Abstract
In this paper, we propose a reinforcement learning-based algorithm for trajectory optimization for constrained dynamical systems. This problem is motivated by the fact that for most robotic systems, the dynamics may not always be known. Generating smooth, dynamically feasible trajectories could be difficult for such systems. Using sampling-based algorithms for motion planning may result in trajectories that are prone to undesirable control jumps. However, they can usually provide a good reference trajectory which a model-free reinforcement learning algorithm can then exploit by limiting the search domain and quickly finding a dynamically smooth trajectory. We use this idea to train a reinforcement learning agent to learn a dynamically smooth trajectory in a curriculum learning setting. Furthermore, for generalization, we parameterize the policies with goal locations, so that the agent can be trained for multiple goals simultaneously. We show result in both simulated environments as well as real experiments, for a 6-DoF manipulator arm operated in position-controlled mode to validate the proposed idea. We compare the proposed ideas against a PID controller which is used to track a designed trajectory in configuration space. Our experiments show that our RL agent trained with a reference path outperformed a model-free PID controller of the type commonly used on many robotic platforms for trajectory tracking.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Learning Robotic Assembly from CAD2018
- Mastering the game of Go without human knowledgeNature, 2017
- Human-level control through deep reinforcement learningNature, 2015
- Reinforcement learning in robotics: A surveyThe International Journal of Robotics Research, 2013
- Synthesis and stabilization of complex behaviors through online trajectory optimization2012
- Fast smoothing of manipulator trajectories using optimal bounded-acceleration shortcuts2010
- Learning and generalization of motor skills by learning from demonstration2009
- Automatic shaping and decomposition of reward functions2007
- Planning Algorithms2006
- Autonomous shaping2006