Causality Detection for Efficient Multi-Agent Reinforcement Learning
Preprint
Abstract
When learning a task as a team, some agents in Multi-Agent Reinforcement Learning (MARL) may fail to understand their true impact in the performance of the team. Such agents end up learning sub-optimal policies, demonstrating undesired lazy behaviours. To investigate this problem, we start by formalising the use of temporal causality applied to MARL problems. We then show how causality can be used to penalise such lazy agents and improve their behaviours. By understanding how their local observations are causally related to the team reward, each agent in the team can adjust their individual credit based on whether they helped to cause the reward or not. We show empirically that using causality estimations in MARL improves not only the holistic performance of the team, but also the individual capabilities of each agent. We observe that the improvements are consistent in a set of different environments.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Neural Granger CausalityIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Cooperative Multi-agent Control Using Deep Reinforcement Learning2017
- Multiagent cooperation and competition with deep reinforcement learningPLOS ONE, 2017
- Multi-agent reinforcement learning as a rehearsal for decentralized planningNeurocomputing, 2016
- Human-level control through deep reinforcement learningNature, 2015
- Investigating Causal Relations by Econometric Models and Cross-spectral MethodsEconometrica, 1969