Sample-efficient multi-agent reinforcement learning with masked reconstruction
Open Access
- 14 September 2023
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 18 (9) , e0291545
Abstract
Deep reinforcement learning (DRL) is a powerful approach that combines reinforcement learning (RL) and deep learning to address complex decision-making problems in high-dimensional environments. Although DRL has been remarkably successful, its low sample efficiency necessitates extensive training times and large amounts of data to learn optimal policies. These limitations are more pronounced in the context of multi-agent reinforcement learning (MARL). To address these limitations, various studies have been conducted to improve DRL. In this study, we propose an approach that combines a masked reconstruction task with QMIX (M-QMIX). By introducing a masked reconstruction task as an auxiliary task, we aim to achieve enhanced sample efficiency—a fundamental limitation of RL in multi-agent systems. Experiments were conducted using the StarCraft II micromanagement benchmark to validate the effectiveness of the proposed method. We used 11 scenarios comprising five easy, three hard, and three very hard scenarios. We particularly focused on using a limited number of time steps for each scenario to demonstrate the improved sample efficiency. Compared to QMIX, the proposed method is superior in eight of the 11 scenarios. These results provide strong evidence that the proposed method is more sample-efficient than QMIX, demonstrating that it effectively addresses the limitations of DRL in multi-agent systems.Keywords
Funding Information
- National Research Foundation of Korea (Brain Korea 21 FOUR)
- National Research Foundation of Korea (Brain Korea 21 FOUR)
This publication has 18 references indexed in Scilit:
- Multi-agent reinforcement learning with approximate model learning for competitive gamesPLOS ONE, 2019
- Emergence of linguistic conventions in multi-agent reinforcement learningPLOS ONE, 2018
- Multiagent cooperation and competition with deep reinforcement learningPLOS ONE, 2017
- Multi-agent reinforcement learning as a rehearsal for decentralized planningNeurocomputing, 2016
- Decentralized Opportunistic Spectrum Resources Access Model and Algorithm toward Cooperative Ad-Hoc NetworksPLOS ONE, 2016
- A Concise Introduction to Decentralized POMDPs2016
- Human-level control through deep reinforcement learningNature, 2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Optimal and Approximate Q-value Functions for Decentralized POMDPsJournal of Artificial Intelligence Research, 2008
- Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents1993