Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning
Authors:
- Tianyi ChenTCTianyi Chen
- Kaiqing ZhangKZKaiqing Zhang
- Georgios B. GiannakisGGGeorgios B. Giannakis
- Tamer BaşarTBTamer Başar
Preprint
Abstract
This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchanges between the learners and the controller are required. For many practical distributed systems, however, the overhead caused by these frequent communication exchanges is considerable, and becomes the bottleneck of the overall performance. To address this challenge, a novel policy gradient approach is developed for solving distributed RL. The novel approach adaptively skips the policy gradient communication during iterations, and can reduce the communication overhead without degrading learning performance. It is established analytically that: i) the novel algorithm has convergence rate identical to that of the plain-vanilla policy gradient; while ii) if the distributed learners are heterogeneous in terms of their reward functions, the number of communication rounds needed to achieve a desirable learning accuracy is markedly reduced. Numerical experiments corroborate the communication reduction attained by the novel algorithm compared to alternatives.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Stochastic Policy Gradient Ascent in Reproducing Kernel Hilbert SpacesIEEE Transactions on Automatic Control, 2020
- Event-triggered communication and control of networked systems for multi-agent consensusAutomatica, 2019
- Communication-Efficient Distributed Statistical InferenceJournal of the American Statistical Association, 2018
- Bandit Convex Optimization for Scalable and Dynamic IoT ManagementIEEE Internet of Things Journal, 2018
- Multiagent cooperation and competition with deep reinforcement learningPLOS ONE, 2017
- Decentralized Stochastic Control with Partial History Sharing: A Common Information ApproachIEEE Transactions on Automatic Control, 2013
- The Complexity of Decentralized Control of Markov Decision ProcessesMathematics of Operations Research, 2002
- Infinite-Horizon Policy-Gradient EstimationJournal of Artificial Intelligence Research, 2001
- Multiagent Systems: A Survey from a Machine Learning PerspectiveAutonomous Robots, 2000
- Optimum Bounds for the Distributions of Martingales in Banach SpacesThe Annals of Probability, 1994