Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations
Authors:
- 1 December 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Abstract
Reinforcement Learning (RL) has achieved significant success in application domains such as robotics, games and health care. However, training RL agents is very time consuming. Current implementations exhibit poor performance due to challenges such as irregular memory accesses and thread-level synchronization overheads on CPU. In this work, we propose a framework for generating scalable reinforcement learning implementations on multi-core systems. Replay Buffer is a key component of RL algorithms which facilitates storage of samples obtained from environmental interactions and data sampling for the learning process. We define a new data structure for Prioritized Replay Buffer based on $K$ -ary sum tree that supports asynchronous parallel insertions, sampling, and priority updates. To address the challenge of irregular memory accesses, we propose a novel data layout to store the nodes of the sum tree that reduces the number of cache misses. Additionally, we propose lazy writing mechanism to reduce thread-level synchronization over-heads of the Replay Buffer operations. Our framework employs parallel actors to concurrently collect data via environmental interactions, and parallel learners to perform stochastic gradient descent using the collected data. Our framework supports a wide range of reinforcement learning algorithms including DQN, DDPG, etc. We demonstrate the effectiveness of our framework in accelerating RL algorithms by performing experiments on CPU + GPU platform using OpenAI benchmarks. Our results show that the performance of our $K$ -ary sum tree based Prioritized Replay Buffer improves the baseline implementations by around 4x~100x. Our proposed synchronization optimizations improve the performance by around 2x~4.4x compared with using a global lock. By plugging our Replay Buffer implementation into existing open source reinforcement learning frameworks, we achieve 1.19x~ 1.75x speedup for various algorithms.
Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Efficient Online Hyperparameter Learning for Traffic Flow Prediction2018
- Mastering the game of Go without human knowledgeNature, 2017
- Customised pearlmutter propagation: A hardware architecture for trust region policy optimisation2017
- In-Datacenter Performance Analysis of a Tensor Processing Unit2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Scaling Distributed Machine Learning with the Parameter Server2014
- MuJoCo: A physics engine for model-based control2012
- MapReduce for Parallel Reinforcement LearningLecture Notes in Computer Science, 2012
- MapReduceCommunications of the ACM, 2008
- Pthreads for Dynamic Parallelism,1998