Optimizing Distributed Reinforcement Learning with Reactor Model and Lingua Franca
Preprint
Abstract
Distributed Reinforcement Learning (RL) frameworks are essential for mapping RL workloads to multiple computational resources, allowing for faster generation of samples, estimation of values, and policy improvement. These computational paradigms require a seamless integration of training, serving, and simulation workloads. Existing frameworks, such as Ray, are not managing this orchestration efficiently. In this study, we've proposed a solution implementing Reactor Model, which enforces a set of actors to have a fixed communication pattern. This allows the scheduler to eliminate works needed for synchronization, such as acquiring and releasing locks for each actor or sending and processing coordination-related messages. Our framework, Lingua Franca (LF), a coordination language based on the Reactor Model, also provides a unified interface that allows users to automatically generate dataflow graphs for distributed RL. On average, LF outperformed Ray in generating samples from OpenAI Gym and Atari environments by 1.21x and 11.62x, reduced the average training time of synchronized parallel Q-learning by 31.2%, and accelerated Multi-Agent RL inference by 5.12x.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- High-performance Deterministic Concurrency Using Lingua FrancaACM Transactions on Architecture and Code Optimization, 2023
- Champion-level drone racing using deep reinforcement learningNature, 2023
- An Extensible, Data-Oriented Architecture for High-Performance, Many-World SimulationACM Transactions on Graphics, 2023
- Model-based Reinforcement Learning: A SurveyFoundations and Trends in Machine Learning, 2023
- A review of cooperative multi-agent deep reinforcement learningApplied Intelligence, 2022
- Toward a Lingua Franca for Deterministic Concurrent SystemsACM Transactions on Embedded Computing Systems, 2021
- Visual Navigation with Asynchronous Proximal Policy Optimization in Artificial AgentsJournal of Robotics, 2020
- Mastering the game of Go with deep neural networks and tree searchNature, 2016