Zhen Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Zhen Liu has a total of 23 co-authors. Here are the top co-authors by number of shared publications:
- Jianqiang Yi
Institute of Automation, Chinese Academy of Sciences, Beijing, China
9 shared publications - Zhiqiang Pu
Institute of Automation, Chinese Academy of Sciences, Beijing, China
9 shared publications - Tianle Zhang
JD Explore Academy
4 shared publications Jianqiang Yi
Chinese Academy of Sciences
3 shared publications- Huimu Wang
Institute of Automation, Chinese Academy of Sciences, Beijing, China
2 shared publications - Jinlin Sun
School of Electrical and Information Engineering, Jiangsu University, Zhenjiang 212013, China
2 shared publications
Latest Publications
- 24 April 2026
- Published by Springer Nature in International Journal of Machine Learning and Cybernetics
- Vol. 17 (6) , 281
State-guided policy learning for effective decision making
- 24 April 2026
- Published by Springer Nature in International Journal of Machine Learning and Cybernetics
- Vol. 17 (6) , 281
State-guided policy learning for effective decision making
- 24 June 2025
- Published by Springer Nature
- p. 182-196
RBHAR: Role-Based Heterogeneous Action Representation in Multi-agent Reinforcement Learning
- 24 June 2025
- Published by Springer Nature
- p. 182-196
RBHAR: Role-Based Heterogeneous Action Representation in Multi-agent Reinforcement Learning
- 4 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (6) , 10736-10748
Cognition-Oriented Multiagent Reinforcement Learning
Inspired by psychological insights into individual behavior, we propose a novel cognition-oriented multiagent reinforcement learning (CORL) framework. CORL equips agents with two distinct types of cognition—situational and self-cognition—derived from local observations. To enhance the informativeness and precision of these cognition types, we introduce two information-theoretical regularizers: one to align situational cognition with the global state and the other to align self-cognition with each agent’s identity for improved role differentiation and team coordination. In addition, the centralized training and decentralized execution framework is adopted to train the policy network. Our simulations demonstrate that CORL effectively harnesses local observations for enriched cooperation, leading to pronounced performance improvements, particularly in challenging tasks.
- 4 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (6) , 10736-10748
Cognition-Oriented Multiagent Reinforcement Learning
Inspired by psychological insights into individual behavior, we propose a novel cognition-oriented multiagent reinforcement learning (CORL) framework. CORL equips agents with two distinct types of cognition—situational and self-cognition—derived from local observations. To enhance the informativeness and precision of these cognition types, we introduce two information-theoretical regularizers: one to align situational cognition with the global state and the other to align self-cognition with each agent’s identity for improved role differentiation and team coordination. In addition, the centralized training and decentralized execution framework is adopted to train the policy network. Our simulations demonstrate that CORL effectively harnesses local observations for enriched cooperation, leading to pronounced performance improvements, particularly in challenging tasks.
- 5 May 2023
- Published by Springer Nature in International Journal of Control, Automation, and Systems
- Vol. 21 (7) , 2350-2362
Robot Subgoal-guided Navigation in Dynamic Crowded Environments with Hierarchical Deep Reinforcement Learning
- 5 May 2023
- Published by Springer Nature in International Journal of Control, Automation, and Systems
- Vol. 21 (7) , 2350-2362
Robot Subgoal-guided Navigation in Dynamic Crowded Environments with Hierarchical Deep Reinforcement Learning
- 10 April 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (9) , 12678-12692
Multiexperience-Assisted Efficient Multiagent Reinforcement Learning
Recently, multiagent reinforcement learning (MARL) has shown great potential for learning cooperative policies in multiagent systems (MASs). However, a noticeable drawback of current MARL is the low sample efficiency, which causes a huge amount of interactions with environment. Such amount of interactions greatly hinders the real-world application of MARL. Fortunately, effectively incorporating experience knowledge can assist MARL to quickly find effective solutions, which can significantly alleviate the drawback. In this article, a novel multiexperience-assisted reinforcement learning (MEARL) method is proposed to improve the learning efficiency of MASs. Specifically, monotonicity-constrained reward shaping is innovatively designed using expert experience to provide additional individual rewards to guide multiagent learning efficiently, with the invariance guarantee of the team optimization objective. Furthermore, a reward distribution estimator is specially developed to model an implicated reward distribution of environment by using transition experience from environment, containing collected samples (state–action pair, reward, and next state). This estimator can predict the expectation reward of each agent for the taken action to accurately estimate the state value function and accelerate its convergence. Besides, the performance of MEARL is evaluated on two multiagent environment platforms: our designed unmanned aerial vehicle combat (UAV-C) and StarCraft II Micromanagement (SCII-M). Simulation results demonstrate that the proposed MEARL can greatly improve the learning efficiency and performance of MASs and is superior to the state-of-the-art methods in multiagent tasks.
- 10 April 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (9) , 12678-12692
Multiexperience-Assisted Efficient Multiagent Reinforcement Learning
Recently, multiagent reinforcement learning (MARL) has shown great potential for learning cooperative policies in multiagent systems (MASs). However, a noticeable drawback of current MARL is the low sample efficiency, which causes a huge amount of interactions with environment. Such amount of interactions greatly hinders the real-world application of MARL. Fortunately, effectively incorporating experience knowledge can assist MARL to quickly find effective solutions, which can significantly alleviate the drawback. In this article, a novel multiexperience-assisted reinforcement learning (MEARL) method is proposed to improve the learning efficiency of MASs. Specifically, monotonicity-constrained reward shaping is innovatively designed using expert experience to provide additional individual rewards to guide multiagent learning efficiently, with the invariance guarantee of the team optimization objective. Furthermore, a reward distribution estimator is specially developed to model an implicated reward distribution of environment by using transition experience from environment, containing collected samples (state–action pair, reward, and next state). This estimator can predict the expectation reward of each agent for the taken action to accurately estimate the state value function and accelerate its convergence. Besides, the performance of MEARL is evaluated on two multiagent environment platforms: our designed unmanned aerial vehicle combat (UAV-C) and StarCraft II Micromanagement (SCII-M). Simulation results demonstrate that the proposed MEARL can greatly improve the learning efficiency and performance of MASs and is superior to the state-of-the-art methods in multiagent tasks.
- 14 October 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 7 (3) , 912-930
Automatic Curriculum Learning for Large-Scale Cooperative Multiagent Systems
Recently, a lot of works have been devoted to researching how agents can learn efficient cooperation in multiagent systems. However, it still remains challenging in large-scale multiagent systems (MASs) due to the complex dynamics between the agents and environment and the dimension explosion of state-action space. In this paper, we propose a novel MultiAgent Automatic Curriculum Learning method (MA-ACL) to solve learning problems of large-scale cooperative MASs by beginning from learning on a multiagent scenario with a few agents and automatically progressively increasing the number of agents. An evaluation mechanism based on self-supervised learning is innovatively designed to automatically generate appropriate curricula with a progressively increasing number of agents. Moreover, since the observation state dimension of agents varies across curricula and the learned policy knowledge needs to be effectively encoded, we design a new Distributed Transferable Relation-modeling Policy network structure (DTRP) to handle the dynamic size of the network input and model relational knowledge between agents and their surrounding environment. Simulation results show that the proposed MA-ACL using DTRP can significantly improve the performance of large-scale multiagent learning compared with manual or non curriculum learning methods, and DTRP greatly boosts the performance of MA-ACL.
- 14 October 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 7 (3) , 912-930
Automatic Curriculum Learning for Large-Scale Cooperative Multiagent Systems
Recently, a lot of works have been devoted to researching how agents can learn efficient cooperation in multiagent systems. However, it still remains challenging in large-scale multiagent systems (MASs) due to the complex dynamics between the agents and environment and the dimension explosion of state-action space. In this paper, we propose a novel MultiAgent Automatic Curriculum Learning method (MA-ACL) to solve learning problems of large-scale cooperative MASs by beginning from learning on a multiagent scenario with a few agents and automatically progressively increasing the number of agents. An evaluation mechanism based on self-supervised learning is innovatively designed to automatically generate appropriate curricula with a progressively increasing number of agents. Moreover, since the observation state dimension of agents varies across curricula and the learned policy knowledge needs to be effectively encoded, we design a new Distributed Transferable Relation-modeling Policy network structure (DTRP) to handle the dynamic size of the network input and model relational knowledge between agents and their surrounding environment. Simulation results show that the proposed MA-ACL using DTRP can significantly improve the performance of large-scale multiagent learning compared with manual or non curriculum learning methods, and DTRP greatly boosts the performance of MA-ACL.
- 8 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 15 (4) , 623-636
Peer Incentive Reinforcement Learning for Cooperative Multiagent Games
Social learning, especially social incentives, is extremely important for humans to achieve a high level of coordination. Inspired by this, we introduce this concept into cooperative multi-agent reinforcement learning (MARL), to implicitly address the credit assignment problem and promote the inter-agent direct interactions for cooperations among agents in cooperative multi-agent games. In this paper, we propose a novel Intrinsic Reward method with Peer Incentives (IRPI) based on actor-critic policy gradient. This method can enable agents to incentivize each other for their cooperations through using causal influence among them. Specifically, a novel intrinsic reward mechanism is innovatively designed to empower each agent the ability to give positive or negative rewards to other peer agents' actions through considering the causal influence of the other agents on it. The mechanism is realized by a feed-forward neural network through utilizing causal influence between the agents. The causal influence of one agent on another is inferred via counterfactual reasoning using the joint action-value function in MARL. The quality of the influence is assessed via counterfactual reasoning using the individual value function in MARL. Simulations are carried out on two popular multi-agent game testbeds: Starcraft II Micromanagement and Multi-Agent Particle Environments. Simulational results demonstrate that the proposed IRPI can enhance cooperations among the agents to achieve better performance compared with a number of state-of-the-art MARL methods in a variety of cooperative multi-agent games.
- 8 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 15 (4) , 623-636
Peer Incentive Reinforcement Learning for Cooperative Multiagent Games
Social learning, especially social incentives, is extremely important for humans to achieve a high level of coordination. Inspired by this, we introduce this concept into cooperative multi-agent reinforcement learning (MARL), to implicitly address the credit assignment problem and promote the inter-agent direct interactions for cooperations among agents in cooperative multi-agent games. In this paper, we propose a novel Intrinsic Reward method with Peer Incentives (IRPI) based on actor-critic policy gradient. This method can enable agents to incentivize each other for their cooperations through using causal influence among them. Specifically, a novel intrinsic reward mechanism is innovatively designed to empower each agent the ability to give positive or negative rewards to other peer agents' actions through considering the causal influence of the other agents on it. The mechanism is realized by a feed-forward neural network through utilizing causal influence between the agents. The causal influence of one agent on another is inferred via counterfactual reasoning using the joint action-value function in MARL. The quality of the influence is assessed via counterfactual reasoning using the individual value function in MARL. Simulations are carried out on two popular multi-agent game testbeds: Starcraft II Micromanagement and Multi-Agent Particle Environments. Simulational results demonstrate that the proposed IRPI can enhance cooperations among the agents to achieve better performance compared with a number of state-of-the-art MARL methods in a variety of cooperative multi-agent games.
- 18 February 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (11) , 8235-8249
Attention Enhanced Reinforcement Learning for Multi agent Cooperation
In this article, a novel method, called attention enhanced reinforcement learning (AERL), is proposed to address issues including complex interaction, limited communication range, and time-varying communication topology for multi agent cooperation. AERL includes a communication enhanced network (CEN), a graph spatiotemporal long short-term memory network (GST-LSTM), and parameters sharing multi-pseudo critic proximal policy optimization (PS-MPC-PPO). Specifically, CEN based on graph attention mechanism is designed to enlarge the agents’ communication range and to deal with complex interaction among the agents. GST-LSTM, which replaces the standard fully connected (FC) operator in LSTM with graph attention operator, is designed to capture the temporal dependence while maintaining the spatial structure learned by CEN. PS-MPC-PPO, which extends proximal policy optimization (PPO) in multi agent systems with parameters’ sharing to scale to environments with a large number of agents in training, is designed with multi-pseudo critics to mitigate the bias problem in training and accelerate the convergence process. Simulation results for three groups of representative scenarios including formation control, group containment, and predator–prey games demonstrate the effectiveness and robustness of AERL.
- 18 February 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (11) , 8235-8249
Attention Enhanced Reinforcement Learning for Multi agent Cooperation
In this article, a novel method, called attention enhanced reinforcement learning (AERL), is proposed to address issues including complex interaction, limited communication range, and time-varying communication topology for multi agent cooperation. AERL includes a communication enhanced network (CEN), a graph spatiotemporal long short-term memory network (GST-LSTM), and parameters sharing multi-pseudo critic proximal policy optimization (PS-MPC-PPO). Specifically, CEN based on graph attention mechanism is designed to enlarge the agents’ communication range and to deal with complex interaction among the agents. GST-LSTM, which replaces the standard fully connected (FC) operator in LSTM with graph attention operator, is designed to capture the temporal dependence while maintaining the spatial structure learned by CEN. PS-MPC-PPO, which extends proximal policy optimization (PPO) in multi agent systems with parameters’ sharing to scale to environments with a large number of agents in training, is designed with multi-pseudo critics to mitigate the bias problem in training and accelerate the convergence process. Simulation results for three groups of representative scenarios including formation control, group containment, and predator–prey games demonstrate the effectiveness and robustness of AERL.
- 5 December 2021
- Published by Springer Nature
- p. 112-124
Semantic Perception Swarm Policy with Deep Reinforcement Learning
- 5 December 2021
- Published by Springer Nature
- p. 112-124
Semantic Perception Swarm Policy with Deep Reinforcement Learning
- Open Access
- 21 March 2021
- Published by MDPI AG in Algorithms
- Vol. 14 (3) , 98
Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
Multiagent cooperation is one of the most attractive research fields in multiagent systems. There are many attempts made by researchers in this field to promote cooperation behavior. However, several issues still exist, such as complex interactions among different groups of agents, redundant communication contents of irrelevant agents, which prevents the learning and convergence of agent cooperation behaviors. To address the limitations above, a novel method called multiagent hierarchical cognition difference policy (MA-HCDP) is proposed in this paper. It includes a hierarchical group network (HGN), a cognition difference network (CDN), and a soft communication network (SCN). HGN is designed to distinguish different underlying information of diverse groups’ observations (including friendly group, enemy group, and object group) and extract different high-dimensional state representations of different groups. CDN is designed based on a variational auto-encoder to allow each agent to choose its neighbors (communication targets) adaptively with its environment cognition difference. SCN is designed to handle the complex interactions among the agents with a soft attention mechanism. The results of simulations demonstrate the superior effectiveness of our method compared with existing methods.
- 21 March 2021
- Published by MDPI AG in Algorithms
- Vol. 14 (3) , 98
Open AccessMultiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
Multiagent cooperation is one of the most attractive research fields in multiagent systems. There are many attempts made by researchers in this field to promote cooperation behavior. However, several issues still exist, such as complex interactions among different groups of agents, redundant communication contents of irrelevant agents, which prevents the learning and convergence of agent cooperation behaviors. To address the limitations above, a novel method called multiagent hierarchical cognition difference policy (MA-HCDP) is proposed in this paper. It includes a hierarchical group network (HGN), a cognition difference network (CDN), and a soft communication network (SCN). HGN is designed to distinguish different underlying information of diverse groups’ observations (including friendly group, enemy group, and object group) and extract different high-dimensional state representations of different groups. CDN is designed based on a variational auto-encoder to allow each agent to choose its neighbors (communication targets) adaptively with its environment cognition difference. SCN is designed to handle the complex interactions among the agents with a soft attention mechanism. The results of simulations demonstrate the superior effectiveness of our method compared with existing methods.