Guided Policy Search for Sequential Multitask Learning
- 19 February 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 49 (1) , 216-226
Abstract
Policy search in reinforcement learning (RL) is a practical approach to interact directly with environments in parameter spaces, that often deal with dilemmas of local optima and real-time sample collection. A promising algorithm, known as guided policy search (GPS), is capable of handling the challenge of training samples using trajectory-centric methods. It can also provide asymptotic local convergence guarantees. However, in its current form, the GPS algorithm cannot operate in sequential multitask learning scenarios. This is due to its batch-style training requirement, where all training samples are collectively provided at the start of the learning process. The algorithm's adaptation is thus hindered for real-time applications, where training samples or tasks can arrive randomly. In this paper, the GPS approach is reformulated, by adapting a recently proposed, lifelong-learning method, and elastic weight consolidation. Specifically, Fisher information is incorporated to impart knowledge from previously learned tasks. The proposed algorithm, termed sequential multitask learning-GPS, is able to operate in sequential multitask learning settings and ensuring continuous policy learning, without catastrophic forgetting. Pendulum and robotic manipulation experiments demonstrate the new algorithms efficacy to learn control policies for handling sequentially arriving training samples, delivering comparable performance to the traditional, and batch-based GPS algorithm. In conclusion, the proposed algorithm is posited as a new benchmark for the real-time RL and robotics research community.Keywords
Funding Information
- National Natural Science Foundation of China (U1613213, 61375005, 61503383, 61210009, 61627808, 91648205, 61702516, 61473236)
- National Key Research and Development Plan of China (2017YFB1300202, 2016YFC0300801)
- Ministry of Science and Technology of the People's Republic of China (2015BAK35B00, 2015BAK35B01)
- Guangdong Provincial Department of Science and Technology (2016B090910001)
- Suzhou Science and Technology Program (SYG201712, SZS201613)
- Strategic Priority Research Program of the Chinese Academy of Science (XDB02080003)
- Key Program Special Fund in XJTLU (KSF-A-01)
- Engineering and Physical Sciences Research Council (EP/M026981/1)
This publication has 14 references indexed in Scilit:
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- Adaptive Control of Robotic Manipulators With Unified Motion ConstraintsIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2016
- Towards real-time parameter optimization for feasible nonlinear control with applications to robot locomotion2016
- Reinforcement learning in robotics: A surveyThe International Journal of Robotics Research, 2013
- MuJoCo: A physics engine for model-based control2012
- A Survey on Policy Search for RoboticsFoundations and Trends in Robotics, 2011
- Learning CPG-based Biped Locomotion with a Policy Gradient Method: Application to a Humanoid RobotThe International Journal of Robotics Research, 2008
- Universal Intelligence: A Definition of Machine IntelligenceMinds and Machines, 2007
- Reinforcement Learning with Replacing Eligibility Traces2007
- Lifelong Learning Algorithms1998