MARRGM: Learning Framework for Multi-Agent Reinforcement Learning via Reinforcement Recommendation and Group Modification
- 16 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (6) , 5385-5392
Abstract
Sample usage efficiency is an important factor affecting the convergence speed of multi-agent deep reinforcement learning (MADRL) algorithms. Most existing experience replay (ER) methods manually select experience samples to update the agent's policy. It is difficult to give suitable and efficient experience samples for different stages of agent policy learning as well as to effectively mine the potential value of experience samples in the replay buffer. Inspired by the idea of recommendation systems, this paper proposes a MADRL framework based on reinforcement recommendation and group modification to improve sample use efficiency and the ability to find the optimal solution of the multi-agent system in different task scenario categories. First, we use the sampling probability of each experience sample output from the recommendation network to recommend sampling instead of manual sampling; simultaneously, we collect the performance of the multi-agent system after updating the policy with the experience sample of recommendation sampling and construct the reinforcement learning process of the recommendation network. Next, we modify the individual policy of the agent according to the group rewards to improve the agent's ability to learn the optimal solution. We then combine and embed the reinforcement recommendation and group modification modules into the MADRL algorithm MAAC. Finally, we experiment with task scenarios, including cooperative collection, command movement, and target navigation, and extend this framework to the MADDPG algorithm to verify its scalability. The experimental results show that the off-policy MADRL algorithms combined with the proposed framework outperform the baseline algorithm in terms of sample usage efficiency and have better universality for the number of agents and scene categories.Keywords
Funding Information
- National Natural Science Foundation of China (62276028, U20A20167)
- Natural Science Foundation of Hebei Province (F202103079)
- Hebei Province Central Leading Local Science and Technology Development (236Z1811G)
- Innovation Capability Improvement Plan Project of Hebei Province (22567626H)
This publication has 10 references indexed in Scilit:
- A survey of multi-agent deep reinforcement learning with communicationAutonomous Agents and Multi-Agent Systems, 2024
- Event-Triggered Model Predictive Control With Deep Reinforcement Learning for Autonomous DrivingIEEE Transactions on Intelligent Vehicles, 2023
- A Deep Reinforcement Learning Framework for Eco-Driving in Connected and Automated Hybrid Electric VehiclesIEEE Transactions on Vehicular Technology, 2023
- A novel robotic grasping method for moving objects based on multi-agent deep reinforcement learningRobotics and Computer-Integrated Manufacturing, 2023
- Multi-agent dueling Q-learning with mean field and value decompositionPattern Recognition, 2023
- Adaptively Calibrated Critic Estimates for Deep Reinforcement LearningIEEE Robotics and Automation Letters, 2022
- Model tree methods for explaining deep reinforcement learning agents in real-time robotic applicationsNeurocomputing, 2022
- Clustering experience replay for the effective exploitation in reinforcement learningPattern Recognition, 2022
- Lighter and Better: Low-Rank Decomposed Self-Attention Networks for Next-Item Recommendation2021
- Lenient Multi-Agent Deep Reinforcement Learning2018