Multi-agent Gradient-Based Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
Author:
Open Access
- 24 June 2024
- journal article
- research article
- Published by Springer Nature in International Journal of Computational Intelligence Systems
- Vol. 17 (1) , 1-18
Abstract
This paper proposes a gradient-based multi-agent actor-critic algorithm for off-policy reinforcement learning using importance sampling. Our algorithm is incremental with full gradients, and its complexity per iteration scales linearly with the size of approximation features. Previous multi-agent actor-critic algorithms are limited to the on-policy setting or off-policy emphatic temporal difference (TD) learning and they do not take advantage of the advances in off-policy gradient temporal difference learning (GTD). As a theoretical contribution, we establish that the critic step of the proposed algorithm converges to the TD solution of the projected Bellman equation and the actor step converges to the set of asymptotically stable fixed points. Numerical experiments on the multi-agent generalization of the Boyan’s chain problem show that the proposed approach provides improved performances in terms of stability and convergence rate as compared with the state-of-the-art baseline algorithm.Keywords
Funding Information
- Wenzhou University (Young Faculty Grant)
- McGill University
This publication has 31 references indexed in Scilit:
- An off-policy multi-agent stochastic policy gradient algorithm for cooperative continuous controlNeural Networks, 2023
- Communication-efficient hierarchical distributed optimization for multi-agent policy evaluationJournal of Computational Science, 2021
- Off-policy learning for adaptive optimal output synchronization of heterogeneous multi-agent systemsAutomatica, 2020
- Multi-agent reinforcement learning for redundant robot control in task-spaceInternational Journal of Machine Learning and Cybernetics, 2020
- Smart generation control based on multi-agent reinforcement learning with the idea of the time tunnelEnergy, 2018
- Cooperative Multi-agent Control Using Deep Reinforcement Learning2017
- Finite-time cluster average consensus for networks via distributed iterationsInternational Journal of Control, Automation, and Systems, 2017
- Finite-time consensus in directed switching network topologies and time-delayed communicationsScientia Iranica, 2011
- Natural actor–critic algorithmsAutomatica, 2009
- Residual Algorithms: Reinforcement Learning with Function Approximation1995