Value-Decomposition Networks For Cooperative Multi-Agent Learning
Preprint
Abstract
We study the problem of cooperative multi-agent reinforcement learning with a single joint reward signal. This class of learning problems is difficult because of the often large combined action and observation spaces. In the fully centralized and decentralized approaches, we find the problem of spurious rewards and a phenomenon we call the "lazy agent" problem, which arises due to partial observability. We address these problems by training individual agents with a novel value decomposition network architecture, which learns to decompose the team value function into agent-wise value functions. We perform an experimental evaluation across a range of partially-observable multi-agent domains and show that learning such value-decompositions leads to superior results, in particular when combined with weight sharing, role information and information channels.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Combining reward shaping and hierarchies for scaling to large multiagent systemsThe Knowledge Engineering Review, 2016
- Potential-based reward shaping for finite horizon online POMDP planningAutonomous Agents and Multi-Agent Systems, 2015
- Multiagent Learning: Basics, Challenges, and ProspectsAI Magazine, 2012
- The world of independent learners is not markovianInternational Journal of Knowledge-based and Intelligent Engineering Systems, 2011
- Optimal and Approximate Q-value Functions for Decentralized POMDPsJournal of Artificial Intelligence Research, 2008
- Analyzing and visualizing multiagent rewards in dynamic and stochastic domainsAutonomous Agents and Multi-Agent Systems, 2008
- A Comprehensive Survey of Multiagent Reinforcement LearningIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2008
- Cooperative Multi-Agent Learning: The State of the ArtAutonomous Agents and Multi-Agent Systems, 2005
- Markov Decision Processes: Discrete Stochastic Dynamic ProgrammingTechnometrics, 1995