An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning
Author:
Preprint
Abstract
Multi-agent reinforcement learning (MARL) has exploded in popularity in recent years. Many approaches have been developed but they can be divided into three main types: centralized training and execution (CTE), centralized training for decentralized execution (CTDE), and Decentralized training and execution (DTE). CTDE methods are the most common as they can use centralized information during training but execute in a decentralized manner -- using only information available to that agent during execution. CTDE is the only paradigm that requires a separate training phase where any available information (e.g., other agent policies, underlying states) can be used. As a result, they can be more scalable than CTE methods, do not require communication during execution, and can often perform well. CTDE fits most naturally with the cooperative case, but can be potentially applied in competitive or mixed settings depending on what information is assumed to be observed. This text is an introduction to CTDE in cooperative MARL. It is meant to explain the setting, basic concepts, and common methods. It does not cover all work in CTDE MARL as the subarea is quite extensive. I have included work that I believe is important for understanding the main concepts in the subarea and apologize to those that I have omitted.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- On Centralized Critics in Multi-Agent Reinforcement LearningJournal of Artificial Intelligence Research, 2023
- Multiagent cooperation and competition with deep reinforcement learningPLOS ONE, 2017
- Multi-agent reinforcement learning as a rehearsal for decentralized planningNeurocomputing, 2016
- A Concise Introduction to Decentralized POMDPs2016
- A Comprehensive Survey of Multiagent Reinforcement LearningIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2008
- Decentralized Control of Cooperative Systems: Categorization and Complexity AnalysisJournal of Artificial Intelligence Research, 2004
- The Complexity of Decentralized Control of Markov Decision ProcessesMathematics of Operations Research, 2002
- Planning and acting in partially observable stochastic domainsArtificial Intelligence, 1998
- Team Decision ProblemsThe Annals of Mathematical Statistics, 1962
- Elements for a Theory of TeamsManagement Science, 1955