Autonomous Dogfight Decision-Making for Air Combat Based on Reinforcement Learning with Automatic Opponent Sampling
Open Access
Abstract
The field of autonomous air combat has witnessed a surge in interest propelled by the rapid progress of artificial intelligence technology. A persistent challenge within this domain pertains to autonomous decision-making for dogfighting, especially when dealing with intricate, high-fidelity nonlinear aircraft dynamic models and insufficient information. In response to this challenge, this paper introduces reinforcement learning (RL) to train maneuvering strategies. In the context of RL for dogfighting, the method by which opponents are sampled assumes significance in determining the efficacy of training. Consequently, this paper proposes a novel automatic opponent sampling (AOS)-based RL framework where proximal policy optimization (PPO) is applied. This approach encompasses three pivotal components: a phased opponent policy pool with simulated annealing (SA)-inspired curriculum learning, an SA-inspired Boltzmann Meta-Solver, and a Gate Function based on the sliding window. The training outcomes demonstrate that this improved PPO algorithm with an AOS framework outperforms existing reinforcement learning methods such as the soft actor–critic (SAC) algorithm and the PPO algorithm with prioritized fictitious self-play (PFSP). Moreover, during testing scenarios, the trained maneuvering policy displays remarkable adaptability when confronted with a diverse array of opponents. This research signifies a substantial stride towards the realization of robust autonomous maneuvering decision systems in the context of modern air combat.Keywords
Funding Information
- Foundation Enhancement Project for Science and Technology (2021-JCJQ-JJ-081, 2024JC-YBQN-0668, 62303489, GKJJ24050502, 2023T160790)
- Natural Science Basic Research Program of Shaanxi (2021-JCJQ-JJ-081, 2024JC-YBQN-0668, 62303489, GKJJ24050502, 2023T160790)
- National Natural Science Foundation of China (2021-JCJQ-JJ-081, 2024JC-YBQN-0668, 62303489, GKJJ24050502, 2023T160790)
- Postdoctoral Science Foundation Special Funding (2021-JCJQ-JJ-081, 2024JC-YBQN-0668, 62303489, GKJJ24050502, 2023T160790)
This publication has 33 references indexed in Scilit:
- Mastering air combat game with deep reinforcement learningDefence Technology, 2024
- Situational continuity-based air combat autonomous maneuvering decision-makingDefence Technology, 2023
- Modeling and Passivity-Based Control for a convertible fixed-wing VTOLApplied Mathematics and Computation, 2023
- Deep Reinforcement Learning-Based Air-to-Air Combat Maneuver Generation in a Realistic EnvironmentIEEE Access, 2023
- Visual Range Maneuver Decision of Unmanned Combat Aerial Vehicle Based on Fuzzy ReasoningInternational Journal of Fuzzy Systems, 2021
- A Dynamic Game Framework for Rational and Persistent Robot Deception With an Application to Deceptive Pursuit-EvasionIEEE Transactions on Automation Science and Engineering, 2021
- Evolving Intelligent System for Trajectory Tracking of Unmanned Aerial VehiclesIEEE Transactions on Automation Science and Engineering, 2021
- Multi-agent hierarchical policy gradient for Air Combat Tactics emergence via self-playEngineering Applications of Artificial Intelligence, 2021
- Application of the epsilon technique to a realistic optimal pursuit-evasion problemJournal of Optimization Theory and Applications, 1975
- A real-time closed-loop solution method for a class of nonlinear differential gamesIEEE Transactions on Automatic Control, 1972