Unmanned Aerial Vehicle Swarm Cooperative Decision-Making for SEAD Mission: A Hierarchical Multiagent Reinforcement Learning Approach
Authors:
Open Access
- 29 August 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 92177-92191
Abstract
Unmanned aerial vehicle (UAV) swarm cooperative decision-making has attracted increasing attentions because of its low-cost, reusable, and distributed characteristics. However, existing non-learning-based methods rely on small-scale, known scenarios, and cannot solve complex multi-agent cooperation problem in large-scale, uncertain scenarios. This paper proposes a hierarchical multi-agent reinforcement learning (HMARL) method to solve the heterogeneous UAV swarm cooperative decision-making problem for the typical suppression of enemy air defense (SEAD) mission, which is decoupled into two sub-problems, i.e., the higher-level target allocation (TA) sub-problem and the lower-level cooperative attacking (CA) sub-problem. A HMARL agent model, consisting of a multi-agent deep Q network (MADQN) based TA agent and multiple independent asynchronous proximal policy optimization (IAPPO) based CA agents, is established. MADQN-TA agent can dynamically adjust the TA schemes according to the relative position. To encourage exploration and promote learning efficiency, the Metropolis criterion and inter-agent information exchange techniques are introduced. IAPPO-CA agent adopts independent learning paradigm, which can easily scale with the number of agents. Comparative simulation results validate the effectiveness, robustness, and scalability of the proposed method.Keywords
Funding Information
- National Natural Science Foundation of China (62106284)
- Natural Science Foundation of Shaanxi province, China (2021JQ-370)
- Young Talent Fund of Association for Science and Technology in Shaanxi, China (20220101)
This publication has 27 references indexed in Scilit:
- Human-level control through deep reinforcement learningNature, 2015
- Real-time trajectory planning for UCAV air-to-surface attack using inverse dynamics optimization method and receding horizon controlChinese Journal of Aeronautics, 2013
- Multilevel Coalition Formation Strategy for Suppression of Enemy Air Defenses MissionsJournal of Aerospace Information Systems, 2013
- Hierarchical reinforcement learning and decision makingCurrent Opinion in Neurobiology, 2012
- Modeling Air Combat by a Moving Horizon Influence Diagram GameJournal of Guidance, Control, and Dynamics, 2006
- Suppression of enemy air defenses (SEAD) as an information duelNaval Research Logistics (NRL), 2002
- Combat UAV real-time SEAD mission simulation1999
- Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learningArtificial Intelligence, 1999
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- Development of a tactical guidance research and evaluation system (TGRES)1989