Distributed Policy Gradient with Heterogeneous Computations for Federated Reinforcement Learning
Authors:
- 22 March 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
The rapid advances in federated learning (FL) in the past few years have recently inspired federated reinforcement learning (FRL), where multiple reinforcement learning (RL) agents collaboratively learn a common decision-making policy without exchanging their raw interaction data with their environments. In this paper, we consider a general FRL framework where agents interact with different environments with identical state and action spaces but different rewards and dynamics. Motivated by the fact that agents often have heterogeneous computation capabilities, we propose a Federated Heterogeneous Policy Gradient (FedHPG) algorithm for FRL, where agents can use different numbers of data trajectories (i.e., batch sizes) and different numbers of local computation iterations for their respective PG algorithms. We characterize performance bounds for the learning accuracy of FedHPG, which shows that it achieves a learning accuracy ∊ with sample complexity of $O$ (1/∊ 2 ), which matches the performance of existing RL algorithms. The results also show the impacts of local iteration numbers and batch sizes for iteration on the learning accuracy. We also extend FedHPG to heterogeneous policy gradient variance reduction (FedHPGVR) algorithm based on the variance reduction method, and analyze the convergence of this algorithm. The theoretical results are verified empirically for benchmark RL tasks.
Keywords
This publication has 8 references indexed in Scilit:
- Federated Transfer Reinforcement Learning for Autonomous Driving2022
- Decentralized multi-agent reinforcement learning with networked agents: recent advancesFrontiers of Information Technology & Electronic Engineering, 2021
- Communication-Efficient Policy Gradient Methods for Distributed Reinforcement LearningIEEE Transactions on Control of Network Systems, 2021
- Challenges of real-world reinforcement learning: definitions, benchmarks and analysisMachine Learning, 2021
- Federated reinforcement learning: techniques, applications, and open challengesIntelligence & Robotics, 2021
- MapReduce for Parallel Reinforcement LearningLecture Notes in Computer Science, 2012
- Infinite-Horizon Policy-Gradient EstimationJournal of Artificial Intelligence Research, 2001
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992