Guoxin Su

Deep reinforcement learning (DRL) has emerged as a powerful paradigm for solving complex decision-making problems. However, DRL-based systems still face significant dependability challenges particularly in real-time environments due to the simulation-to-reality gap, out-of-distribution observations, and the critical impact of latency. Latency-induced faults, in particular, can lead to unsafe or unstable behaviour, yet existing fault-tolerance approaches to DRL systems lack formal methods to rigorously analyse and optimise performance and safety simultaneously in real-time settings. To address this, we propose a formal framework for designing and analysing real-time switching mechanisms between DRL agents and alternative controllers. Our approach leverages Timed Automata (TAs) for explicit switch logic design, which is then syntactically converted to a Markov Decision Process (MDP) for formal analysis. We develop a novel convex query technique for multi-objective model checking, enabling the optimisation of soft performance objectives while ensuring hard safety constraints for MDPs. Furthermore, we present MOPMC, a GPU-accelerated software tool implementing this technique, demonstrating superior scalability in both model size and objective numbers.

Due to its exceptional learning ability, multi-agent deep reinforcement learning (MADRL) has garnered widespread research interest. However, since the learning is data-driven and involves sampling from millions of steps, training a large number of agents is inherently challenging and inefficient. Inspired by the human learning process, we aim to transfer knowledge from humans to avoid starting from scratch. Given the growing emphasis on the Human-on-the-Loop concept, this study focuses on addressing the challenges of large-population learning by incorporating suboptimal human knowledge into the cooperative multi-agent environment. To leverage human experience, we integrate human knowledge into the training process of MADRL, representing it in natural language rather than specific action-state pairs. Compared to previous works, we further consider the attributes of transferred knowledge to assess its impact on algorithm scalability. Additionally, we examine several features of knowledge mapping to effectively convert human knowledge to the action space where agent learning occurs. In reaction to the disparity in knowledge construction between humans and agents, our approach allows agents to decide freely which portions of the state space to leverage human knowledge. From the challenging domains of the StarCraft Multi-agent Challenge, our method successfully alleviates the scalability issue in MADRL. Furthermore, we find that, despite individual-type knowledge significantly accelerating the training process, cooperative-type knowledge is more desirable for addressing a large agent population. We hope this study provides valuable insights into applying and mapping human knowledge, ultimately enhancing the interpretability of agent behavior.

Multiagent reinforcement learning (MARL) has garnered extensive research attention due to its strong learning capabilities, leading to its deployment in increasingly challenging scenarios. Although progress has been made toward more generalizable solutions, many MARL algorithms continue to struggle with balancing scalability and heterogeneity, particularly under conditions of growing uncertainty. Research has shown that combining dense local interactions with sparse global interactions can significantly enhance scalability while preserving agent heterogeneity. Motivated by these insights and inspired by human social behavior, we propose a novel hierarchical method that integrates human guidance with multiagent systems (MASs). Rather than requiring agents to learn from scratch, our method transfers abstract knowledge from humans, employing fuzzy logic to manage the inherent uncertainty in this guidance and reduce the required human effort. To accommodate both local and global interactions, we introduce two levels of human guidance: individual action guidance for agents and an attention graph to describe agent relationships. Our proposed approach is end-to-end and compatible with diverse MARL algorithms. We evaluate our approach in the starcraft multiagent challenge (SMAC) and SMACv2 environments. Empirical results demonstrate its effectiveness, even under low-performance fuzzy human guidance.