Multi-agent Deep Reinforcement Learning with Extremely Noisy Observations
Preprint
Abstract
Multi-agent reinforcement learning systems aim to provide interacting agents with the ability to collaboratively learn and adapt to the behaviour of other agents. In many real-world applications, the agents can only acquire a partial view of the world. Here we consider a setting whereby most agents' observations are also extremely noisy, hence only weakly correlated to the true state of the environment. Under these circumstances, learning an optimal policy becomes particularly challenging, even in the unrealistic case that an agent's policy can be made conditional upon all other agents' observations. To overcome these difficulties, we propose a multi-agent deep deterministic policy gradient algorithm enhanced by a communication medium (MADDPG-M), which implements a two-level, concurrent learning mechanism. An agent's policy depends on its own private observations as well as those explicitly shared by others through a communication medium. At any given point in time, an agent must decide whether its private observations are sufficiently informative to be shared with others. However, our environments provide no explicit feedback informing an agent whether a communication action is beneficial, rather the communication policies must also be learned through experience concurrently to the main policies. Our experimental results demonstrate that the algorithm performs well in six highly non-stationary environments of progressively higher complexity, and offers substantial performance gains compared to the baselines.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Multiagent cooperation and competition with deep reinforcement learningPLOS ONE, 2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Human-level control through deep reinforcement learningNature, 2015
- Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problemsThe Knowledge Engineering Review, 2012
- A multi-agent solution to energy management in hybrid renewable energy generation systemRenewable Energy, 2011
- A Multiagent Approach to Autonomous Intersection ManagementJournal of Artificial Intelligence Research, 2008
- Self-improving reactive agents based on reinforcement learning, planning and teachingMachine Learning, 1992