YDYali Du

Yali Du

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Yali Du has a total of 110 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Learning safe behaviour via justified human preferences and hypothetical queries

    by Ilias Kazantzidis, Timothy J. Norman, Yali Du, Christopher T. Freeman

    Although reinforcement learning is a powerful paradigm for agent sequential decision-making, it cannot be used in its traditional form in most safety-critical environments. Human feedback can enable an agent to learn a good policy while avoiding unsafe states, but at the cost of human time. We present JPAL-HA, a model for safe learning in safety-critical environments that is grounded on two novel ideas: (i) human preferences over a choice of actions are augmented with justifications such as one action is preferred because the other is unsafe; and (ii) we use these justifications to guide the generation of future queries over hypothetical actions to enable the agent to more effectively map out unsafe regions. We evaluate JPAL-HA using modified configurations of the Island Navigation environment from the AI Safety Gridworlds. We empirically show that it significantly improves safety compared to existing approaches for safe learning from human preferences and outperforms other safe human-in-the-loop algorithms in terms of sample efficiency. We also include an analysis of safety/human burden trade-off. Additionally, we evaluate our method with people training a mobile robot, demonstrating that a diverse group of people can complete training in only a few minutes and that hypothetical actions not only further improve safety, but also slightly decrease the training time and keep it to admissible levels.

  • Open Access

    Evaluating Collective Behaviour of Hundreds of LLM Agents

    by Richard Willis, Jianing Zhao, Yali Du, Joel Z. Leibo

    As autonomous agents powered by LLM are increasingly deployed in society, understanding their collective behaviour in social dilemmas becomes critical. We introduce an evaluation framework where LLMs generate strategies encoded as algorithms, enabling inspection prior to deployment and scaling to populations of hundreds of agents -- substantially larger than in previous work. We find that more recent models tend to produce worse societal outcomes compared to older models when agents prioritise individual gain over collective benefits. Using cultural evolution to model user selection of agents, our simulations reveal a significant risk of convergence to poor societal equilibria, particularly when the relative benefit of cooperation diminishes and population sizes increase. We release our code as an evaluation suite for developers to assess the emergent collective behaviour of their models.

  • Open Access

    Is Pure Exploitation Sufficient in Exogenous MDPs with Linear Function Approximation?

    by Hao Liang, Jiayu Cheng, Sean R. Sinclair, Yali Du

    Exogenous MDPs (Exo-MDPs) capture sequential decision-making where uncertainty comes solely from exogenous inputs that evolve independently of the learner's actions. This structure is especially common in operations research applications such as inventory control, energy storage, and resource allocation, where exogenous randomness (e.g., demand, arrivals, or prices) drives system behavior. Despite decades of empirical evidence that greedy, exploitation-only methods work remarkably well in these settings, theory has lagged behind: all existing regret guarantees for Exo-MDPs rely on explicit exploration or tabular assumptions. We show that exploration is unnecessary. We propose Pure Exploitation Learning (PEL) and prove the first general finite-sample regret bounds for exploitation-only algorithms in Exo-MDPs. In the tabular case, PEL achieves $\widetilde{O}(H^2|Ξ|\sqrt{K})$. For large, continuous endogenous state spaces, we introduce LSVI-PE, a simple linear-approximation method whose regret is polynomial in the feature dimension, exogenous state space, and horizon, independent of the endogenous state and action spaces. Our analysis introduces two new tools: counterfactual trajectories and Bellman-closed feature transport, which together allow greedy policies to have accurate value estimates without optimism. Experiments on synthetic and resource-management tasks show that PEL consistently outperforming baselines. Overall, our results overturn the conventional wisdom that exploration is required, demonstrating that in Exo-MDPs, pure exploitation is enough.

  • Towards Communication Efficient Multi-Agent Cooperations: Reinforcement Learning and LLM

    by Yang Su, Yali Du, Yansha Deng, Mischa Dohler

    Multi-agent systems (MAS) have gained popularity due to their effectiveness in diverse applications. Among these, decentralized approaches, which rely on inter-agent communication, have emerged as a promising alternative to centralized MAS coordination methods. However, existing communication protocols often learn abstract representations that are difficult for humans to interpret. Also, the limited wireless bandwidth in real-world applications imposes a significant challenge on efficient communication among agents. To address these issues, in this paper, we consider a multi-robot navigation scenario and propose a goal-oriented human-interpretable feature communication protocol, enabling robots to transmit only the most semantically important features. Building on this protocol, we first introduce a Multi-Agent Proximal Policy Optimization (MAPPO)-based Bandwidth-Adaptive Dual-Policy (MAPPO BDP) framework, which employs a dual-policy architecture to jointly optimize feature selection and navigation under bandwidth constraints. However, as MAPPO-BDP requires retraining for different bandwidth constraints, we further propose a Large Language Model (LLM)-based Bandwidth-Adaptive Dual-Policy (LLM-BDP) framework. By leveraging LLM-based multi-agents, LLM-BDP can adapt to varying bandwidth constraints by simply modifying the input prompt, thereby eliminating the need for re training. Experimental results show that both MAPPO-BDP and LLM-BDP frameworks successfully complete navigation tasks within a limited number of steps by effectively communicating the most important features under bandwidth constraints. We demonstrate that LLM-BDP achieves better reasoning adapt ability by utilizing more advanced LLMs and engineering example prompts. Notably, with human-interpretable feature communication protocol, robots trained with the MAPPO-BDP framework can seamlessly collaborate with robots from the LLM BDP framework to complete tasks, even without co-training, demonstrating the practical applicability of the proposed communication protocol in new scenarios.


  • Open Access

    Self-Verifying Reflection Helps Transformers with CoT Reasoning

    by Zhongwei Yu, Wannian Xia, Xue Yan, Bo Xu, Haifeng Zhang, Yali Du, Jun Wang

    Advanced large language models (LLMs) frequently reflect in reasoning chain-of-thoughts (CoTs), where they self-verify the correctness of current solutions and explore alternatives. However, given recent findings that LLMs detect limited errors in CoTs, how reflection contributes to empirical improvements remains unclear. To analyze this issue, in this paper, we present a minimalistic reasoning framework to support basic self-verifying reflection for small transformers without natural language, which ensures analytic clarity and reduces the cost of comprehensive experiments. Theoretically, we prove that self-verifying reflection guarantees improvements if verification errors are properly bounded. Experimentally, we show that tiny transformers, with only a few million parameters, benefit from self-verification in both training and reflective execution, reaching remarkable LLM-level performance in integer multiplication and Sudoku. Similar to LLM results, we find that reinforcement learning (RL) improves in-distribution performance and incentivizes frequent reflection for tiny transformers, yet RL mainly optimizes shallow statistical patterns without faithfully reducing verification errors. In conclusion, integrating generative transformers with discriminative verification inherently facilitates CoT reasoning, regardless of scaling and natural language.

  • Open Access

    PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments

    by Olivier Schipper, Yudi Zhang, Yali Du, Mykola Pechenizkiy, Meng Fang

    LLM-based agents have shown promise in various cooperative and strategic reasoning tasks, but their effectiveness in competitive multi-agent environments remains underexplored. To address this gap, we introduce PillagerBench, a novel framework for evaluating multi-agent systems in real-time competitive team-vs-team scenarios in Minecraft. It provides an extensible API, multi-round testing, and rule-based built-in opponents for fair, reproducible comparisons. We also propose TactiCrafter, an LLM-based multi-agent system that facilitates teamwork through human-readable tactics, learns causal dependencies, and adapts to opponent strategies. Our evaluation demonstrates that TactiCrafter outperforms baseline approaches and showcases adaptive learning through self-play. Additionally, we analyze its learning process and strategic evolution over multiple game episodes. To encourage further research, we have open-sourced PillagerBench, fostering advancements in multi-agent AI for competitive environments.

  • Open Access

    Quantifying the Self-Interest Level of Markov Social Dilemmas

    by Richard Willis, Yali Du, Joel Z. Leibo, Michael Luck

    This paper introduces a novel method for estimating the self-interest level of Markov social dilemmas. We extend the concept of self-interest level from normal-form games to Markov games, providing a quantitative measure of the minimum reward exchange required to align individual and collective interests. We demonstrate our method on three environments from the Melting Pot suite, representing either common-pool resources or public goods. Our results illustrate how reward exchange can enable agents to transition from selfish to collective equilibria in a Markov social dilemma. This work contributes to multi-agent reinforcement learning by providing a practical tool for analysing complex, multistep social dilemmas. Our findings offer insights into how reward structures can promote or hinder cooperation, with potential applications in areas such as mechanism design.


  • Open Access

    Abstract Counterfactuals for Language Model Agents

    by Edoardo Pona, Milad Kazemi, Yali Du, David Watson, Nicola Paoletti

    Counterfactual inference is a powerful tool for analysing and evaluating autonomous agents, but its application to language model (LM) agents remains challenging. Existing work on counterfactuals in LMs has primarily focused on token-level counterfactuals, which are often inadequate for LM agents due to their open-ended action spaces. Unlike traditional agents with fixed, clearly defined action spaces, the actions of LM agents are often implicit in the strings they output, making their action spaces difficult to define and interpret. Furthermore, the meanings of individual tokens can shift depending on the context, adding complexity to token-level reasoning and sometimes leading to biased or meaningless counterfactuals. We introduce \emph{Abstract Counterfactuals}, a framework that emphasises high-level characteristics of actions and interactions within an environment, enabling counterfactual reasoning tailored to user-relevant features. Our experiments demonstrate that the approach produces consistent and meaningful counterfactuals while minimising the undesired side effects of token-level methods. We conduct experiments on text-based games and counterfactual text generation, while considering both token-level and latent-space interventions.

  • Open Access

    Will Systems of LLM Agents Cooperate: An Investigation into a Social Dilemma

    by Richard Willis, Yali Du, Joel Z Leibo, Michael Luck

    As autonomous agents become more prevalent, understanding their collective behaviour in strategic interactions is crucial. This study investigates the emergent cooperative tendencies of systems of Large Language Model (LLM) agents in a social dilemma. Unlike previous research where LLMs output individual actions, we prompt state-of-the-art LLMs to generate complete strategies for iterated Prisoner's Dilemma. Using evolutionary game theory, we simulate populations of agents with different strategic dispositions (aggressive, cooperative, or neutral) and observe their evolutionary dynamics. Our findings reveal that different LLMs exhibit distinct biases affecting the relative success of aggressive versus cooperative strategies. This research provides insights into the potential long-term behaviour of systems of deployed LLM-based autonomous agents and highlights the importance of carefully considering the strategic environments in which they operate.


  • Open Access

    NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning

    by Wei Liu, Siya Qi, Xinyu Wang, Chen Qian, Yali Du, Yulan He

    Wei Liu, Siya Qi, Xinyu Wang, Chen Qian, Yali Du, Yulan He. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.