Zhiliang Fan

Sparse rewards remain a major obstacle to efficient reinforcement learning (RL), particularly in long-horizon goal-reaching tasks where little feedback is available before successful termination. This difficulty is further complicated by time-limited trajectories, for which the absence of observed success indicates incomplete follow-up rather than necessarily a failed attempt. Existing progress-based shaping methods generally do not explicitly model this distinction, which can bias temporal progress estimates and weaken the resulting shaping signal. To address this issue, we propose Censoring-Aware Time-to-Success Potential Shaping (CTPS), a reward-shaping method that treats successful trajectories as observed time-to-event data and time-limited trajectories as right-censored data. CTPS fits a discrete hazard model to estimate the conditional time-to-success distribution, summarizes the resulting survival curve using the restricted mean time-to-success, and converts this estimate into a bounded potential. The learned temporal signal is then incorporated through a potential-based reward difference, with the potential held fixed during each target-update interval. Experiments show that CTPS reduces RMST estimation error by approximately 93% relative to timeout-as-event estimation under heavy censoring, while improving sparse-maze success rates by approximately 157% over NaiveTTS-PBRS. It also preserves state-ranking correlations above 0.98 under the evaluated behavior-policy shifts. These results characterize the temporal-estimation performance of CTPS and its use as a shaping signal in the evaluated sparse-reward tasks.

Adversarial policies provide a practical means of stress-testing reinforcement-learning agents by inducing failure through environment interactions without modifying the deployed victim policy. Learning effective adversarial policies under partial observability is challenging because the attacker must act from incomplete observations without access to victim-side parameters, gradients, or internal states. Existing approaches often rely on instantaneous observability patterns or predefined partitions, which can discard temporal interaction information and provide limited guidance for exploring rarely encountered but potentially vulnerable contexts. We propose CAAP (Context-Adaptive Adversarial Policy), which is a black-box adversarial policy framework that represents recent interaction history with a recurrent Context Encoder, combines multiple action-value experts through a differentiable Soft Policy Router, and encourages early exploration through Context Coverage Exploration using an annealed KNN-based coverage bonus in the learned context space. Context Coverage Exploration is used only during training, leaving the Context Encoder, Soft Policy Router, and action-value experts as the deployed policy. In the primary particle-pursuit benchmark, CAAP achieves the lowest in-domain catch rate among the five internal methods and the lowest macro-average catch rate across eight zero-update transfer conditions. These results support the use of history-dependent Soft Policy Routing with training-only Context Coverage Exploration for black-box adversarial policy learning under partial observability.

Cooperative operations in which a UAV advances ahead of a UGV to conduct forward reconnaissance are critical in disaster relief and urban inspection missions. Prevalent air–ground coordination methods operate under the assumption of ideal communication or treat connectivity as a secondary objective. However, obstacle occlusion, such as high-rise buildings in urban areas and mountainous terrain, results in Non-Line-of-Sight (NLOS) conditions, disrupting communication between the two platforms. To address these challenges, this paper introduces a cooperative control framework based on dynamically varying modulation matrices for both the UAV and the UGV. By evaluating and mapping occlusion risks in real time, the cooperative motions of the UAV and UGV are adaptively adjusted to maintain Line-of-Sight (LOS). An LOS assessment function is designed and mapped to the eigenvalues of the modulation matrices, enabling smooth and adaptive coordination under changing environmental conditions while avoiding the limitations of traditional discrete mode-switching strategies. Theoretical analysis and simulation results confirm that the proposed approach not only ensures stable LOS connectivity but also enhances trajectory smoothness, adaptability, and computational efficiency.

Navigation spoofing has been widely utilized in unmanned aircraft vehicle (UAV) countermeasures, due to its advantages of covertness, effectiveness, and dynamic trajectory control ability. However, existing research faces two primary challenges. Firstly, sudden changes in the target UAV’s trajectory can result in a significant degradation in the spoofing performance, which may enable the onboard inertial components to detect and identify the ongoing spoofing attempts. Secondly, gradual accumulation of control errors over time degenerates the spoofing effect. To address these problems, we propose a dynamic trajectory spoofing approach for UAVs based on model predictive control (MPC), which progressively steers the UAVs towards the predetermined trajectory of the spoofer. Simulation results demonstrate a substantial enhancement in dynamic trajectory control performance and decrease in accumulation error compared to the existing methods.