Hui Li

Artificial intelligence for aircraft guidance is a hot research topic, and deep reinforcement learning is one of the promising methods. However, due to the different movement patterns of destinations in different guidance tasks, it is inefficient to train agents from scratch. In this article, a policy-reuse algorithm based on destination position prediction is proposed to solve this problem. First, the reward function is optimized to improve flight trajectory quality and training efficiency. Then, by predicting the possible termination position of the destinations in different moving patterns, the problem is transformed into a fixed-position destination aircraft guidance problem. Last, taking the agent in the fixed-position destination scenario as the baseline agent, a new guidance agent can be trained efficiently. Simulation results show that this method can significantly improve the training efficiency of agents in new tasks, and its performance is stable in tasks with different similarities. This research broadens the application scope of the policy-reuse approach and also enlightens the research in other fields.

Deep reinforcement learning (DRL) has been widely adopted recently for its ability to solve decision-making problems that were previously out of reach due to a combination of nonlinear and high dimensionality. In the last few years, it has spread in the field of air traffic control (ATC), particularly in conflict resolution. In this work, we conduct a detailed review of existing DRL applications for conflict resolution problems. This survey offered a comprehensive review based on segments as (1) fundamentals of conflict resolution, (2) development of DRL, and (3) various applications of DRL in conflict resolution classified according to environment, model, algorithm, and evaluating indicator. Finally, an open discussion is provided that potentially raises a range of future research directions in conflict resolution using DRL. The objective of this review is to present a guidance point for future research in a more meaningful direction.

Finding Nash equilibrium in the domain of imperfect information games as a challenging problem has received much attention. Neural Fictitious Self-Play (NFSP) is a popular model-free machine learning algorithm and has computed approximate Nash equilibrium on such games. However, the deep reinforcement learning method used to approximate the best response in NFSP requires reaching a fully observable Markov state, while the states in imperfect information games are partially observable and non-Markovian, which results in a poor approximation of the best response. Thus, NFSP needs more iterations to converge. In this study, we present a new reinforcement learning method that is inspired by counterfactual regret minimization to relax the Markov requirement by iteratively updating policy according to the regret matching process. Combining this new reinforcement learning algorithm with fictitious play, we further present a novel algorithm to find approximate Nash equilibrium in zero-sum imperfect information games. Experimental results in three benchmark games show that this new algorithm can find approximate Nash equilibrium effectively and converge much faster compared with baseline.

The unmanned aerial vehicle (UAV) has huge application potential in disaster rescue, natural detection, and military reconnaissance. In these scenarios, task assignment is essential and facing the problem of NP-hard, and as the scale of the Multi-UAV system increase, it is harder to solve the problem. In this paper, a CSCM-SA (Cosine Similarity Clustering Method- Simulated Annealing) is proposed. Firstly, the cosine similarity clustering method is used for pre-clustering of large-scale targets; Secondly, simulated annealing is employed to optimize routers for each UAV; Finally, to cope with the instability of CSCM results, two strategies of employing genetic algorithm are studied. Simulation results show that the CSCM-SA algorithm is effective to conduct task assignments for the large-scale multi-UAV systems.

Multiagent reinforcement learning holds considerable promise to deal with cooperative multiagent tasks. Unfortunately, the only global reward shared by all agents in the cooperative tasks may lead to the lazy agent problem. To cope with such a problem, we propose a generating individual intrinsic reward algorithm, which introduces an intrinsic reward encoder to generate an individual intrinsic reward for each agent and utilizes the hypernetworks as the decoder to help to estimate the individual action values of the decomposition methods based on the generated individual intrinsic reward. Experimental results in the StarCraft II micromanagement benchmark prove that the proposed algorithm can increase learning efficiency and improve policy performance.

Reinforcement learning holds considerable promise to help address sequential decision-making problems, in which Q-learning is one of the most used algorithms. However, Q-learning suffers from overestimation errors, especially when action values in the same state are similar. To reduce such damages, we introduce an internal supplemental action measurement based on the variation of the expected state values between a state transition, which can measure the action causing the state transition. For the reason that the internal supplemental action measurement can increase or decrease the action values according to the action performance, it can increase the gap of the action values, thus reducing the sensitivity to the overestimation error. The experimental results in the Markov chain and the video games demonstrate the performance advantage of applying the internal supplemental action measurement, in which the mean evaluating scores with the internal supplemental action measurement are 131.6% in SpaceInvaders, 187.9% in Seaquest, and 176.6% in Asterix respectively of that without the internal supplemental action measurement.

To enhance the performance of guiding an aircraft to a moving destination in a certain direction in three-dimensional continuous space, it is essential to develop an efficient intelligent algorithm. In this article, a pretrained proximal policy optimization (PPO) with reward shaping algorithm, which does not require an accurate model, is proposed to solve the guidance problem of manned aircraft and unmanned aerial vehicles. Continuous action reward function and position reward function are presented, by which the training speed is increased and the performance of the generated trajectory is improved. Using pretrained PPO, a new agent can be trained efficiently for a new task. A reinforcement learning framework is built, in which an agent can be trained to generate a reference trajectory or a series of guidance instructions. General simulation results show that the proposed method can significantly improve the training efficiency and trajectory performance. The carrier-based aircraft approach simulation is carried out to prove the application value of the proposed approach.

In a one-on-one air combat game, the opponent’s maneuver strategy is usually not deterministic, which leads us to consider a variety of opponent’s strategies when designing our maneuver strategy. In this paper, an alternate freeze game framework based on deep reinforcement learning is proposed to generate the maneuver strategy in an air combat pursuit. The maneuver strategy agents for aircraft guidance of both sides are designed in a flight level with fixed velocity and the one-on-one air combat scenario. Middleware which connects the agents and air combat simulation software is developed to provide a reinforcement learning environment for agent training. A reward shaping approach is used, by which the training speed is increased, and the performance of the generated trajectory is improved. Agents are trained by alternate freeze games with a deep reinforcement algorithm to deal with nonstationarity. A league system is adopted to avoid the red queen effect in the game where both sides implement adaptive strategies. Simulation results show that the proposed approach can be applied to maneuver guidance in air combat, and typical angle fight tactics can be learnt by the deep reinforcement learning agents. For the training of an opponent with the adaptive strategy, the winning rate can reach more than 50%, and the losing rate can be reduced to less than 15%. In a competition with all opponents, the winning rate of the strategic agent selected by the league system is more than 44%, and the probability of not losing is about 75%.

The primary objective of this study is to incorporate the deep reinforcement learning (DRL) technique in conflict detection and resolution (CD&R) control strategies to generate an optimised trajectory for air traffic controllers as reference, in order to improve efficiency and reduce the amount of heading angle change. A DRL environment which can be applied to CD&R agent training is developed. The agent receives the current state of multiple aircrafts in a sector and generates an action to change the heading angle of an aircraft to avoid conflict. A K ‐Control Actor‐Critic algorithm is proposed to limit the number of control times and a two‐dimensional continuous action selection policy is utilised. The simulation results show the feasibility of DRL applied in CD&R and there is an obvious advantage in computational efficiency.