Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
Preprint
Abstract
Many complex multi-agent systems such as robot swarms control and autonomous vehicle coordination can be modeled as Multi-Agent Reinforcement Learning (MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge (SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX, allowing for performance improvement in SMAC. In this paper, we investigate the code-level optimizations of these variants and the monotonicity constraint. (1) We find that such improvements of the variants are significantly affected by various code-level optimizations. (2) The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC; (3) beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency in SMAC and DEPP. We also discuss why monotonicity constraints work well in purely cooperative tasks with a theoretical analysis. We open-source the code at \url{https://github.com/hijkzzz/pymarl2}.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Counterfactual Multi-Agent Policy GradientsProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Multi-agent reinforcement learning as a rehearsal for decentralized planningNeurocomputing, 2016
- An Overview of Recent Progress in the Study of Distributed Multi-Agent CoordinationIEEE Transactions on Industrial Informatics, 2012
- Coordinated Multi-Agent Reinforcement Learning in Networked Distributed POMDPsProceedings of the AAAI Conference on Artificial Intelligence, 2011