Neural networks and differential dynamic programming for reinforcement learning problems
- 1 May 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 26, 5434-5441
Abstract
We explore a model-based approach to reinforcement learning where partially or totally unknown dynamics are learned and explicit planning is performed. We learn dynamics with neural networks, and plan behaviors with differential dynamic programming (DDP). In order to handle complicated dynamics, such as manipulating liquids (pouring), we consider temporally decomposed dynamics. We start from our recent work [1] where we used locally weighted regression (LWR) to model dynamics. The major contribution of this paper is making use of deep learning in the form of neural networks with stochastic DDP, and showing the advantages of neural networks over LWR. For this purpose, we extend neural networks for: (1) modeling prediction error and output noise, (2) computing an output probability distribution for a given input distribution, and (3) computing gradients of output expectation with respect to an input. Since neural networks have nonlinear activation functions, these extensions were not easy. We provide an analytic solution for these extensions using some simplifying assumptions. We verified this method in pouring simulation experiments. The learning performance with neural networks was better than that of LWR. The amount of spilled materials was reduced. We also present early results of robot experiments using a PR2. Accompanying video: https://youtu.be/aM3hE1J5W98.Keywords
This publication has 16 references indexed in Scilit:
- Pouring Skills with Planning and Learning Modeled from Human DemonstrationsInternational Journal of Humanoid Robotics, 2015
- Learning contact-rich manipulation skills with guided policy search2015
- DeepPose: Human Pose Estimation via Deep Neural Networks2014
- DCOB: Action space for reinforcement learning of high DoF robotsAutonomous Robots, 2013
- Reinforcement learning to adjust parametrized motor primitives to new situationsAutonomous Robots, 2012
- Inverse Kinematics Solver for Android Faces with Elastic Skin2012
- Policy search for motor primitives in roboticsMachine Learning, 2010
- The CMA Evolution Strategy: A Comparing Review2007
- Neural Fitted Q Iteration – First Experiences with a Data Efficient Neural Reinforcement Learning Method2005
- A Second-order Gradient Method for Determining Optimal Trajectories of Non-linear Discrete-time SystemsInternational Journal of Control, 1966