STAS: Spatial-Temporal Return Decomposition for Solving Sparse Rewards Problems in Multi-agent Reinforcement Learning
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (16) , 17337-17345
Abstract
Centralized Training with Decentralized Execution (CTDE) has been proven to be an effective paradigm in cooperative multi-agent reinforcement learning (MARL). One of the major challenges is credit assignment, which aims to credit agents by their contributions. They lack the functionality to model complicated relations of the delayed global reward in the temporal dimension and suffer from inefficiencies. To tackle this, we introduce Spatial-Temporal Attention with Shapley (STAS), a novel method that learns credit assignment in both temporal and spatial dimensions. It first decomposes the global return back to each time step, then utilizes the Shapley Value to redistribute the individual payoff from the decomposed global reward. To mitigate the computational complexity of the Shapley Value, we introduce an approximation of marginal contribution and utilize Monte Carlo sampling to estimate it. We evaluate our method on an Alice & Bob example and MPE environments across different scenarios. Our results demonstrate that our method effectively assigns spatial-temporal credit, outperforming all state-of-the-art baselines.Keywords
This publication has 12 references indexed in Scilit:
- Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning2021
- Multi-agent reinforcement learning for redundant robot control in task-spaceInternational Journal of Machine Learning and Cybernetics, 2020
- Shapley Q-Value: A Local Reward Approach to Solve Global Reward GamesProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Grandmaster level in StarCraft II using multi-agent reinforcement learningNature, 2019
- Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward2018
- Self-Attention with Relative Position Representations2018
- Long short-term memory recurrent neural network architectures for large scale acoustic modeling2014
- A Comprehensive Survey of Multiagent Reinforcement LearningIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2008
- Cooperative Multi-Agent Learning: The State of the ArtAutonomous Agents and Multi-Agent Systems, 2005
- The Shapley value on convex geometriesDiscrete Applied Mathematics, 2000