Hierarchical Reinforcement Learning for Concurrent Discovery of Compound and Composable Policies
Authors:
- 1 November 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1818-1825
Abstract
A common strategy to deal with the expensive reinforcement learning (RL) of complex tasks is to decompose them into a collection of subtasks that are usually simpler to learn as well as reusable for new problems. However, when a robot learns the policies for these subtasks, common approaches treat every policy learning process separately. Therefore, all these individual (composable) policies need to be learned before tackling the learning process of the complex task through policies composition. Moreover, such composition of individual policies is usually performed sequentially, which is not suitable for tasks that require to perform the subtasks concurrently. In this paper, we propose to combine a set of composable Gaussian policies corresponding to these subtasks using a set of activation vectors, resulting in a complex Gaussian policy that is a function of the means and covariances matrices of the composable policies. Moreover, we propose an algorithm for learning both compound and composable policies within the same learning process by exploiting the off-policy data generated from the compound policy. The algorithm is built on a maximum entropy RL approach to favor exploration during the learning process. The results of the experiments show that the experience collected with the compound policy permits not only to solve the complex task but also to obtain useful composable policies that successfully perform in their corresponding subtasks.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Composable Modular Reinforcement LearningProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Hierarchical reinforcement learning of multiple grasping strategies with human instructionsAdvanced Robotics, 2018
- Composable Deep Reinforcement Learning for Robotic Manipulation2018
- Hierarchical Deep Reinforcement Learning for Continuous Action ControlIEEE Transactions on Neural Networks and Learning Systems, 2018
- Development of a human size and strength compliant bi-manual platform for realistic heavy manipulation tasks2017
- Robot Learning2016
- Multiobjective Reinforcement Learning: A Comprehensive OverviewIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2014
- Combining learned controllers to achieve new goals based on linearly solvable MDPs2014
- Recent Advances in Hierarchical Reinforcement LearningDiscrete Event Dynamic Systems, 2003
- Multiple Model-Based Reinforcement LearningNeural Computation, 2002