PEARL: Enabling Portable, Productive, and High-Performance Deep Reinforcement Learning using Heterogeneous Platforms
Open Access
- 7 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 41-50
Abstract
Deep Reinforcement Learning (DRL) is vital in various AI applications. DRL algorithms comprise diverse compute kernels, which may not be simultaneously optimized using a homogeneous architecture. However, even with available heterogeneous architectures, optimizing DRL performance remains a challenge due to the complexity of hardware and programming models employed in modern data centers. To address this, we introduce PEARL, a toolkit for composing parallel DRL systems on heterogeneous platforms consisting of general-purpose processors (CPUs) and accelerators (GPUs, FPGAs). Our innovations include: 1. A general training protocol agnostic of the underlying hardware, enabling portable implementations across various platforms. 2. Incorporation of DRL-specific optimizations on runtime scheduling and resource allocation, facilitating parallelized training and enhancing the overall system performance. 3. Automatic optimization of DRL task-to-device assignments through throughput estimation. 4. High-level API for productive development using the toolkit. We showcase our toolkit through experimentation with two widely used DRL algorithms, DQN and DDPG, on two diverse heterogeneous platforms. The generated implementations outperform state-of-the-art libraries for CPU-GPU platforms by up to 2.2× throughput improvements, and 2.4× higher performance portability across platforms.Keywords
Funding Information
- NSF (National Science Foundation) (2009057)
This publication has 13 references indexed in Scilit:
- A Framework for Mapping DRL Algorithms With Prioritized Replay Buffer Onto Heterogeneous PlatformsIEEE Transactions on Parallel and Distributed Systems, 2023
- Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations2021
- Scientific Computing With Python on High-Performance Heterogeneous SystemsComputing in Science & Engineering, 2021
- Navigating Performance, Portability, and ProductivityComputing in Science & Engineering, 2021
- ThunderGP2021
- Study of sample efficiency improvements for reinforcement learning algorithms2020
- Accelerating Proximal Policy Optimization on CPU-FPGA Heterogeneous Platforms2020
- FA3C2019
- Black-box data-efficient policy search for robotics2017
- Accelerating a Random Forest Classifier: Multi-Core, GP-GPU, or FPGA?2012