CuMARL: Curiosity-Based Learning in Multiagent Reinforcement Learning
Open Access
- 16 August 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 87254-87265
Abstract
In this paper, we propose a novel curiosity-based learning algorithm for Multi-agent Reinforcement Learning (MARL) to attain efficient and effective decision-making. We employ the centralized training with decentralized execution framework (CTDE) and consider that each agent has knowledge of the prior action distribution of others. To quantify the difference in agents’ knowledge, curiosity, we introduce conditional mutual information (CMI) regularization and use the amount of information for updating decision-making policy. Then, to deploy these learning frameworks in a large-scale MARL setting while acquiring high sample efficiency, we consider a Kullback-Leibler (KL) divergence-based prioritization of experiences. We evaluate the effectiveness of the proposed algorithm in three different levels of StarCraft Multi Agent Challenge (SMAC) scenarios using the PyMARL framework. The simulation-based performance analysis shows that the proposed technique significantly improves the test win rate compared to various state-of-the-art MARL benchmarks, such as the Optimistically Weighted Monotonic Value Function Factorization (OW_QMIX) and Learning Individual Intrinsic Reward (LIIR).Keywords
Funding Information
- Electronics and Telecommunications Research Institute (ETRI) funded by the Korean Government
- Core Technology Research for Self-Improving Integrated Artificial Intelligence System (22ZS1100)
- Exploratory Research for Human-Agent Collaboration in Multi-Agent Games (19YE1410)
- AFOSR (FA2386-19-1-4020)
This publication has 14 references indexed in Scilit:
- An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing PoliciesProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Multi-Agent Actor-Critic with Hierarchical Graph Attention NetworkProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Skill-based curiosity for intrinsically motivated reinforcement learningMachine Learning, 2019
- Experience Replay Optimization2019
- Attention-Based Curiosity in Multi-Agent Reinforcement Learning Environments2019
- Curiosity-Driven Exploration by Self-Supervised Prediction2017
- Intrinsically motivated model learning for developing curious robotsArtificial Intelligence, 2017
- Intrinsically Motivated Reinforcement Learning: An Evolutionary PerspectiveIEEE Transactions on Autonomous Mental Development, 2010
- SIBLING RELATIONSHIP QUALITY: Its Causes and ConsequencesAnnual Review of Psychology, 1998
- Why Siblings Are Important Agents of Cognitive Development: A Comparison of Siblings and PeersChild Development, 1993