Decentralized Multi-Agents by Imitation of a Centralized Controller
Authors:
- Alex Tong LinALAlex Tong Lin
- Mark J. DebordMDMark J. Debord
- Katia EstabridisKEKatia Estabridis
- Gary HewerGHGary Hewer
- Guido MontufarGMGuido Montufar
- Stanley OsherSOStanley Osher
Preprint
Abstract
We consider a multi-agent reinforcement learning problem where each agent seeks to maximize a shared reward while interacting with other agents, and they may or may not be able to communicate. Typically the agents do not have access to other agent policies and thus each agent is situated in a non-stationary and partially-observable environment. In order to obtain multi-agents that act in a decentralized manner, we introduce a novel algorithm under the popular framework of centralized training, but decentralized execution. This training framework first obtains solutions to a multi-agent problem with a single centralized joint-space learner, which is then used to guide imitation learning for independent decentralized multi-agents. This framework has the flexibility to use any reinforcement learning algorithm to obtain the expert as well as any imitation learning algorithm to obtain the decentralized agents. This is in contrast to other multi-agent learning algorithms that, for example, can require more specific structures. We present some theoretical bounds for our method, and we show that one can obtain decentralized solutions to a multi-agent problem through imitation learning.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Decentralization of Multiagent Policies by Learning What to Communicate2019
- An Algorithmic Perspective on Imitation LearningFoundations and Trends in Robotics, 2018
- Emergence of Grounded Compositional Language in Multi-Agent PopulationsProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Counterfactual Multi-Agent Policy GradientsProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Mastering the game of Go without human knowledgeNature, 2017
- Multi-agent reinforcement learning as a rehearsal for decentralized planningNeurocomputing, 2016
- Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problemsThe Knowledge Engineering Review, 2012
- Hysteretic Q-learning : an algorithm for Decentralized Reinforcement Learning in Cooperative Multi-Agent Teams2007
- Cooperative Multi-Agent Learning: The State of the ArtAutonomous Agents and Multi-Agent Systems, 2005
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement LearningMachine Learning, 1992