Ziyou Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Ziyou Zhang has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Qianchuan Zhao
Tsinghua University
4 shared publications - Dianyu Zhong
Center for Intelligent and Networked Systems (CFINS), Department of Automation and BNRist, Tsinghua University, Beijing, China
2 shared publications - Kailai Sun
Singapore-MIT Alliance for Research and Technology Centre (SMART) Singapore
2 shared publications - Bo Xu
Institute of Automation, Chinese Academy of Sciences, Beijing, China
1 shared publication - Fa-An Dai
Center for Intelligent and Networked Systems (CFINS), Department of Automation and BNRist, Tsinghua University, Beijing, China
1 shared publication - Jian Kang
Institute for Environmental Design and Engineering, The Bartlett, University College London, London, UK
1 shared publication
Latest Publications
- 5 August 2025
- Published by Elsevier in Energy and Buildings
- Vol. 347
Topology-aware hypergraph reinforcement learning for indoor occupant-centric HVAC control
- 5 August 2025
- Published by Elsevier in Energy and Buildings
- Vol. 347
Topology-aware hypergraph reinforcement learning for indoor occupant-centric HVAC control
- 22 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (7) , 6896-6903
Maximum Next-State Entropy for Efficient Reinforcement Learning
Entropy regularization is widely used to improve policy optimization and encourage exploration in reinforcement learning. By maximizing both the expected return and entropy, the agent aims to succeed at the task while acting as randomly as possible. However, current methods based on policy entropy encourage the agent to explore diverse actions, but they do not directly promote exploring diverse states. In this study, we theoretically reveal the challenge of optimizing the agent's next-state entropy and the gap between maximum next-state entropy and current policy entropy regularization methods. To address this limitation, we introduce Maximum Next-State Entropy (MNSE), a novel method that maximizes next-state entropy through an action mapping layer following the inner policy. We provide a theoretical analysis demonstrating that MNSE can maximize next-state entropy by optimizing the entropy of the inner policy. We conduct extensive experiments on various continuous control tasks and demonstrate that MNSE can significantly improve the performance of RL algorithms.
- 22 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (7) , 6896-6903
Maximum Next-State Entropy for Efficient Reinforcement Learning
Entropy regularization is widely used to improve policy optimization and encourage exploration in reinforcement learning. By maximizing both the expected return and entropy, the agent aims to succeed at the task while acting as randomly as possible. However, current methods based on policy entropy encourage the agent to explore diverse actions, but they do not directly promote exploring diverse states. In this study, we theoretically reveal the challenge of optimizing the agent's next-state entropy and the gap between maximum next-state entropy and current policy entropy regularization methods. To address this limitation, we introduce Maximum Next-State Entropy (MNSE), a novel method that maximizes next-state entropy through an action mapping layer following the inner policy. We provide a theoretical analysis demonstrating that MNSE can maximize next-state entropy by optimizing the entropy of the inner policy. We conduct extensive experiments on various continuous control tasks and demonstrate that MNSE can significantly improve the performance of RL algorithms.
- 1 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (7) , 4259-4266
A Learning-Based Method for Computing Control Barrier Functions of Nonlinear Systems With Control Constraints
Verifying the safety of states and designing a safety controller are very important for safety critical systems (for example, robotic and automotive systems). Since the reachable set of a state is hard to calculate online, it is difficult to determine whether the current state will enter the unsafe set in the future. Thus, the safe set of a system is usually obtained by calculating the forward invariant set offline. For control affine systems, the control barrier function provides a linear safety constraint on the control, which helps us obtain safety control online. However, the computation of control barrier functions is recognized as difficult, and existing methods make over-approximations or are only applicable to problems without control constraints. In this letter, we propose a learning-based method to compute the control barrier function for nonlinear control affine systems with control constraints. The method consists of a constraint satisfaction (CS) algorithm and a set expansion (SE) algorithm. The CS algorithm computes the control barrier function that satisfies the constraints, and the SE algorithm expands the volume of the safe set. The numerical results demonstrate that our algorithm outperforms existing methods and can improve existing control barrier functions.
- 1 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (7) , 4259-4266
A Learning-Based Method for Computing Control Barrier Functions of Nonlinear Systems With Control Constraints
Verifying the safety of states and designing a safety controller are very important for safety critical systems (for example, robotic and automotive systems). Since the reachable set of a state is hard to calculate online, it is difficult to determine whether the current state will enter the unsafe set in the future. Thus, the safe set of a system is usually obtained by calculating the forward invariant set offline. For control affine systems, the control barrier function provides a linear safety constraint on the control, which helps us obtain safety control online. However, the computation of control barrier functions is recognized as difficult, and existing methods make over-approximations or are only applicable to problems without control constraints. In this letter, we propose a learning-based method to compute the control barrier function for nonlinear control affine systems with control constraints. The method consists of a constraint satisfaction (CS) algorithm and a set expansion (SE) algorithm. The CS algorithm computes the control barrier function that satisfies the constraints, and the SE algorithm expands the volume of the safe set. The numerical results demonstrate that our algorithm outperforms existing methods and can improve existing control barrier functions.
- 23 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automatic Control
- Vol. 68 (6) , 3830-3837
A Warm-Start Strategy in Interior Point Methods for Shrinking Horizon Model Predictive Control With Variable Discretization Step
In this article, we present a warm-start point algorithm in interior point methods for shrinking horizon model predictive control with a variable discretization step. In the algorithm, a convex combination of components of the earlier optimal solution is used to construct an interpolation point, and we use the convex combination of the modified interpolation point and the cold-start point to obtain a warm-start point. We prove that the worst-case iteration complexity of our strategy is better than that of the cold-start. In the numerical experiment of fuel-optimal planetary powered-descent guidance problems, our strategy reduces the number of iterations of second-order cone programming by about 80% with the number of samples available in practical applications.
- 23 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automatic Control
- Vol. 68 (6) , 3830-3837
A Warm-Start Strategy in Interior Point Methods for Shrinking Horizon Model Predictive Control With Variable Discretization Step
In this article, we present a warm-start point algorithm in interior point methods for shrinking horizon model predictive control with a variable discretization step. In the algorithm, a convex combination of components of the earlier optimal solution is used to construct an interpolation point, and we use the convex combination of the modified interpolation point and the cold-start point to obtain a warm-start point. We prove that the worst-case iteration complexity of our strategy is better than that of the cold-start. In the numerical experiment of fuel-optimal planetary powered-descent guidance problems, our strategy reduces the number of iterations of second-order cone programming by about 80% with the number of samples available in practical applications.