He Bai

We propose a systematic approach to leverage structures in inter-agent couplings for efficient model-free cooperative multi-agent reinforcement learning (MARL). We model the cooperative MARL problem via a Bayesian network and characterize the subset of agents, termed as the value dependency set, whose information is required by each agent to estimate its local action value function exactly. Moreover, we propose a partially decentralized training decentralized execution (P-DTDE) paradigm based on the value dependency set, and establish that the total variance of the P-DTDE policy gradient estimator is less than the centralized training decentralized execution (CTDE) policy gradient estimator. We derive a cooperative multi-agent policy gradient theorem based on the P-DTDE scheme and develop a scalable actor-critic algorithm. We demonstrate the efficiency and scalability of the proposed algorithm on multi-warehouse resource allocation and multi-zone temperature control examples. For dense value dependency sets, we propose an approximation scheme based on truncation of the Bayesian network and empirically show that it achieves a faster convergence than the exact value dependence set for applications with a large number of agents.

Wind estimation can be widely used in the planning and control of uncrewed aerial systems and for meteorology and environmental studies. Fusion of wind estimates from multiple quadcopters can result in improved accuracy of wind estimation that may not be attainable by a single quadcopter. In this article, we consider a multi-quadcopter wind field estimation problem. We design an invariant extended Kalman filter (IEKF) and integrate it with a sequential covariance intersection (SCI) algorithm to fuse wind estimates from multiple quadcopters. We also develop baseline algorithms such as a least-squares method and a multiplicative-EKF (MEKF) SCI algorithm. The effectiveness of the proposed IEKF-SCI algorithm is compared with the baseline algorithms and demonstrated in simulations under various wind field models.

Wind estimation for small unmanned aerial vehicles (sUAV) can improve their navigation and flight performance and be used in environmental studies, agriculture, and meteorology. In this paper, we consider a nonlinear estimation problem where a quadcopter hovers in a wind field. Based on a general form of system dynamics including drag and thrust models and IMU biases, we establish the invariance of system dynamics and equivariance of measurements by designing transformations to states, inputs, and measurements. We then design an invariant extended Kalman filter (IEKF) to estimate the quadcopter's states and wind velocities. We conduct five outdoor experiments to validate the effectiveness of the designed IEKF. The experimental results show that the IEKF performs the best when compared to benchmark methods, such as a static mapping method, an EKF, and a translational EKF (TEKF). Particularly, the IEKF produces better transient performance when compared to the EKF.

Approximate methods to solve stochastic optimal control (SOC) problems have received significant interest from researchers in the past decade. Probabilistic inference approaches to SOC have been developed to solve nonlinear quadratic Gaussian problems. In this work, we propose an Expectation-Maximization (EM) based inference procedure to generate state-feedback controls for constrained SOC problems. We consider the inequality constraints for the state and controls and also the structural constraints for the controls. We employ barrier functions to address state and control constraints. We show that the expectation step leads to smoothing of the state-control pair while the the maximization step on the non-zero subsets of the control parameters allows inference of structured stochastic optimal controllers. We demonstrate the effectiveness of the algorithm on unicycle obstacle avoidance, four-unicycle formation control, and quadcopter navigation in windy environment examples. In these examples, we perform an empirical study on the parametric effect of barrier functions on the state constraint satisfaction. We also present a comparative study of smoothing algorithms on the performance of the proposed approach.

In a mixed-traffic environment, where autonomous vehicles (AVs) and human-driven vehicles are operating side by side, cooperative driving can become a useful tool to facilitate safe coordination. Unlike AVs, human-driven vehicles are difficult to model and control. Existing literature is largely focused on collaboration between AVs considering human vehicles as obstacles or unconnected and uncontrollable entities. In this paper, we present a cooperative driving framework that actively influences human-driven vehicles via advising commands and enables them to cooperate with AVs to achieve coordinated driving behaviors. We formulate cooperative driving as a stochastic model predictive control (sMPC) problem and consider human drivers' various stochastic aspects, such as attentiveness and tendency to follow an advisory action. The solutions to the sMPC provide advisory for longitudinal actions and lane change to human-driven vehicles and control commands to autonomous vehicles. With simulation and human-in-the-loop experimental results, we examine human drivers' reactions in complex driving scenarios and demonstrate the effectiveness of the developed method.

Developing scalable and efficient reinforcement learning algorithms for cooperative multi-agent control has received significant attention over the past years. Existing literature has proposed inexact decompositions of local Q-functions based on empirical information structures between the agents. In this paper, we exploit inter-agent coupling information and propose a systematic approach to exactly decompose the local Q-function of each agent. We develop an approximate least square policy iteration algorithm based on the proposed decomposition and identify two architectures to learn the local Q-function for each agent. We establish that the worst-case sample complexity of the decomposition is equal to the centralized case and derive necessary and sufficient graphical conditions on the inter-agent couplings to achieve better sample efficiency. We demonstrate the improved sample efficiency and computational efficiency on numerical examples.

Achieving distributed reinforcement learning (RL) for large-scale cooperative multi-agent systems (MASs) is challenging because: (i) each agent has access to only limited information; (ii) issues on convergence or computational complexity emerge due to the curse of dimensionality. In this paper, we propose a general computationally efficient distributed framework for cooperative multi-agent reinforcement learning (MARL) by utilizing the structures of graphs involved in this problem. We introduce three coupling graphs describing three types of inter-agent couplings in MARL, namely, the state graph, the observation graph and the reward graph. By further considering a communication graph, we propose two distributed RL approaches based on local value-functions derived from the coupling graphs. The first approach is able to reduce sample complexity significantly under specific conditions on the aforementioned four graphs. The second approach provides an approximate solution and can be efficient even for problems with dense coupling graphs. Here there is a trade-off between minimizing the approximation error and reducing the computational complexity. Simulations show that our RL algorithms have a significantly improved scalability to large-scale MASs compared with centralized and consensus-based distributed RL algorithms.

Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale networks. In this paper, we propose a novel distributed zeroth-order algorithm by leveraging the network structure inherent in the optimization objective, which allows each agent to estimate its local gradient by local cost evaluation independently, without use of any consensus protocol. The proposed algorithm exhibits an asynchronous update scheme, and works for stochastic nonconvex optimization with a possibly nonconvex feasible domain based on the block coordinate descent method. The algorithm is later employed as a distributed model-free RL algorithm for distributed linear quadratic regulator design. We provide an empirical validation of the proposed algorithm to benchmark its performance on convergence rate and variance against a centralized ZOO algorithm.

Autonomous vehicles are gradually entering the transportation system. The traffic will become more heterogeneous since both autonomous and human-driven vehicles will share the roads. Cooperative driving, by promoting synchronized actions and shared situational awareness among vehicles, can significantly enhance driving safety. On the other hand, understanding human drivers is a pivotal step for cooperative driving in such mixed traffic environments, which facilitates effective interaction between human drivers and their vehicles. This paper presents a testbed that can be used to conduct research in intelligent and cooperative driving. The testbed consists of driving simulators, custom-designed copilots with an Artificial Intelligence engine, an optimization server, and a cloud database. The copilot is capable of sensing and understanding the human driver, the vehicle and the traffic. It can assist the driver by providing timely alerts on potential risks. Most importantly, it can communicate with other nearby vehicles for cooperative driving. Two case studies are presented to validate and evaluate the testbed. The first case study demonstrates the performance of the copilot in human distraction detection and driving assistance. The second case study focuses on cooperative driving between one human-driven vehicle and two connected autonomous vehicles in a lane-changing scenario. We expect this research testbed to be used in various research projects that involve human-driven vehicles and connected autonomous vehicles.

Achieving distributed reinforcement learning (RL) for large-scale cooperative multi-agent systems (MASs) is challenging because: (i) each agent has access to only limited information; (ii) issues on scalability and sample efficiency emerge due to the curse of dimensionality. In this paper, we propose a general distributed framework for sample efficient cooperative multi-agent reinforcement learning (MARL) by utilizing the structures of graphs involved in this problem. We introduce three coupling graphs describing three types of inter-agent couplings in MARL, namely, the state graph, observation graph and reward graph. By further considering a communication graph, we propose two distributed RL approaches based on local value functions derived from the coupling graphs. The first approach is able to reduce sample complexity significantly under specific conditions on the aforementioned four graphs. The second approach provides an approximate solution and can be efficient even for problems with dense coupling graphs. Here there is a trade-off between minimizing the approximation error and reducing the computational complexity. Simulations show that our RL algorithms have a significantly improved scalability to large-scale MASs compared with centralized and consensus-based distributed RL algorithms.