Remembering the Markov Property in Cooperative MARL
Preprint
Abstract
Cooperative multi-agent reinforcement learning (MARL) is typically formalised as a Decentralised Partially Observable Markov Decision Process (Dec-POMDP), where agents must reason about the environment and other agents' behaviour. In practice, current model-free MARL algorithms use simple recurrent function approximators to address the challenge of reasoning about others using partial information. In this position paper, we argue that the empirical success of these methods is not due to effective Markov signal recovery, but rather to learning simple conventions that bypass environment observations and memory. Through a targeted case study, we show that co-adapting agents can learn brittle conventions, which then fail when partnered with non-adaptive agents. Crucially, the same models can learn grounded policies when the task design necessitates it, revealing that the issue is not a fundamental limitation of the learning models but a failure of the benchmark design. Our analysis also suggests that modern MARL environments may not adequately test the core assumptions of Dec-POMDPs. We therefore advocate for new cooperative environments built upon two core principles: (1) behaviours grounded in observations and (2) memory-based reasoning about other agents, ensuring success requires genuine skill rather than fragile, co-adapted agreements.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning2023
- The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games2022
- The Hanabi challenge: A new frontier for AI researchArtificial Intelligence, 2020
- Multi-agent reinforcement learning as a rehearsal for decentralized planningNeurocomputing, 2016
- Mutual Information between Discrete and Continuous Data SetsPLOS ONE, 2014
- Nonapproximability Results for Partially Observable Markov Decision ProcessesJournal of Artificial Intelligence Research, 2001