Combining learned controllers to achieve new goals based on linearly solvable MDPs
Authors:
- 1 May 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10504729,p. 5252-5259
Abstract
Learning complicated behaviors usually involves intensive manual tuning and expensive computational optimization because we have to solve a nonlinear Hamilton-Jacobi-Bellman (HJB) equation. Recently, Todorov proposed a class of the so-called Linearly solvable Markov Decision Process (LMDP) which converts a nonlinear HJB equation to a linear differential equation. Linearity of the simplified HJB equation allows us to apply superposition to derive a new composite controller from a set of learned primitive controllers. However, his method was a model-based approach and it was not evaluated in a real domain. This study proposes a model-free method which is similar to the Least Squares Temporal Difference (LSTD) learning. In this method, the exponentially transformed cost function can be regarded as the discount factor in LSTD. Our proposed method is applied to learning walking behaviors with the quadruped robot to evaluate in real robot experiments. The goal of each primitive task is to go to the specific target position in the environment and that of the composite task is to approach arbitrary region represented by the primitives' target positions. Experimental results show that the composite policy can be used as a good initial policy for the new task.Keywords
This publication has 20 references indexed in Scilit:
- Reinforcement learning with state-dependent discount factor2013
- Evaluation of linearly solvable Markov decision process with dynamic model learning in a mobile robot navigation taskFrontiers in Neurorobotics, 2013
- Motor primitive discovery2012
- Model learning for robot control: a surveyCognitive Processing, 2011
- Model-Based Influences on Humans' Choices and Striatal Prediction ErrorsNeuron, 2011
- Linear Bellman combination for control of character animationACM Transactions on Graphics, 2009
- Efficient computation of optimal actionsProceedings of the National Academy of Sciences of the United States of America, 2009
- Reinforcement learning in the brainJournal of Mathematical Psychology, 2009
- Technical Update: Least-Squares Temporal Difference LearningMachine Learning, 2002
- Multiple Model-Based Reinforcement LearningNeural Computation, 2002