Stabilizing Multi-Agent Deep Reinforcement Learning by Implicitly Estimating Other Agents’ Behaviors
Authors:
- 1 May 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3547-3551
Abstract
Deep reinforcement learning (DRL) is able to learn control policies for many complicated tasks, but it’s power has not been unleashed to handle multi-agent circumstances. Independent learning, where each agent treats others as part of the environment and learns its own policy without considering others’ policies is a simple way to apply DRL to multi-agent tasks. However, since agents’ policies change as learning proceeds, from the perspective of each agent, the environment is non-stationary, which makes conventional DRL methods inefficient. To cope with this challenge, we propose a novel approach where each agent uses an implicit estimate of others’ actions to guide its own policy learning. We demonstrate that given the implicit estimate of others’ actions, each agent can learn its policy in a relatively stationary environment. Extensive experiments show that our method significantly alleviates the non-stationarity and outperforms the state-of-the-art in terms of both convergence speed and policy performance.Keywords
This publication has 7 references indexed in Scilit:
- Efficient Multi-agent Cooperative Navigation in Unknown Environments with Interlaced Deep Reinforcement Learning2019
- Cooperative Multi-agent Control Using Deep Reinforcement Learning2017
- Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates2017
- Multiagent cooperation and competition with deep reinforcement learningPLOS ONE, 2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Human-level control through deep reinforcement learningNature, 2015
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998