Yongzhe Chang

Deep Multi-Agent Reinforcement Learning (MARL) excels in co-operative tasks but often struggles with local optima in high - dimensional joint action spaces. In contrast, Evolutionary Algorithms (EAs) offer robust global exploration capabilities. Although hybrid approaches seek to combine the strengths of both paradigms, they typically face a critical bottleneck: the prohibitive sample cost of evaluating large populations via environment rollouts. To address this challenge, we propose Surrogate-assisted Evolutionary Multi-Agent Reinforcement Learning (SEMARL), a unified framework that synergizes gradient-based refinement with surrogate-assisted evolutionary search. SEMARL employs a cooperative co-evolutionary architecture to maintain diverse agent policies and injects gradient-refined parameters into the population to accelerate convergence. Crucially, we leverage the centralized critic from the gradient learner as a computationally efficient surrogate for fitness estimation. To prevent misleading guidance from an inaccurate critic, we introduce an adaptive reliability control mechanism based on temporal difference (TD) error, which dynamically regulates the surrogate's influence. Experiments on the Multi-Agent MuJoCo benchmark demonstrate that SEMARL significantly outperforms other algorithms, achieving superior asymptotic performance with substantially higher sample efficiency.

Real-world decision-making systems often suffer from non-negligible observation and action delays arising from sensing and actuation, which break the standard Markov assumption and degrade reinforcement learning (RL) performance. We present Delay-resilient Encoder-Enhanced RL (DEER), a practical framework that systematically integrates well-established techniques (sequence-to-sequence representation learning, information-state construction, and offline-to-online RL) to enable robust performance in delayed environments. DEER first pretrains an encoder on offline delay-free trajectories and then uses it online to map delayed observations with recent action histories into fixed-length context vectors for standard RL agents. We further provide a theoretical decomposition that links reconstruction error, distributional mismatch, latent-policy generalization, and environment mismatch to overall performance, offering interpretable guidance for design choices. Empirically, integrating DEER with Soft Actor-Critic yields competitive and stable performance across a range of constant and random delaysettings on six continuous-control benchmarks. The contribution is therefore twofold: (i) a practical, deployable engineering solution for delayed RL that leverages existing building blocks effectively; and (ii) a quantitative analysis that explains when and why this integration works.