Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning
- 6 May 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Control of Network Systems
- Vol. 9 (2) , 917-929
Abstract
This article deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multiagent reinforcement learning (RL) and parallel RL , where frequent information exchanges between the learners and the controller are required. For many practical distributed systems, however, the overhead caused by these frequent communication exchanges is considerable, and becomes the bottleneck of the overall performance. To address this challenge, a novel policy gradient approach is developed for solving distributed RL. The novel approach adaptively skips the policy gradient communication during iterations, and can reduce the communication overhead without degrading learning performance. It is established analytically that: i) the novel algorithm has a convergence rate identical to that of the plain-vanilla policy gradient; while ii) if the distributed learners are heterogeneous in terms of their reward functions, the number of communication rounds needed to achieve a desirable learning accuracy is markedly reduced. Numerical experiments corroborate the communication reduction attained by the novel algorithm compared to alternatives.Keywords
All Related Versions
Funding Information
- National Science Foundation (1 509 040, 1 508 993, 1 711 471)
- Army Research Laboratory (W911NF-17-2-0196)
- Rensselaer-IBM AI Research Collaboration
- IBM AI Horizons Network
This publication has 17 references indexed in Scilit:
- Asynchronous periodic event-triggered coordination of multi-agent systems2017
- Cooperative Mission Planning for Multi-UAV Teams2014
- ${{\cal Q} {\cal D}}$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through ${\rm Consensus} + {\rm Innovations}$IEEE Transactions on Signal Processing, 2013
- Decentralized Stochastic Control with Partial History Sharing: A Common Information ApproachIEEE Transactions on Automatic Control, 2013
- MapReduce for Parallel Reinforcement LearningLecture Notes in Computer Science, 2012
- Distributed Event-Triggered Control for Multi-Agent SystemsIEEE Transactions on Automatic Control, 2011
- The Complexity of Decentralized Control of Markov Decision ProcessesMathematics of Operations Research, 2002
- General principles of learning-based multi-agent systems1999
- Q-learningMachine Learning, 1992
- The strength of weak learnabilityMachine Learning, 1990