Synthesizing Control Barrier Functions With Feasible Region Iteration for Safe Reinforcement Learning
- 23 November 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automatic Control
- Vol. 69 (4) , 2713-2720
Abstract
Safety is a critical concern when applying reinforcement learning (RL) to real-world control problems. A widely used method for ensuring safety is to learn a control barrier function with heuristic feasibility labels that come from expert demonstrations [1] or constraint functions [2]. However, their forward invariant sets fall short of the maximum feasible region because of inaccurate labels. This paper proposes an algorithm called feasible region iteration (FRI) that learns the maximum feasible region to generate accurate feasibility labels. The core of FRI is a constraint decay function (CDF), which comes with a self-consistency condition and naturally leads to the constraint Bellman equation. The optimal CDF, which represents the maximum feasible region, is learned through the iteration of feasible region identification and feasible region expansion. Experiment results show that our algorithm achieves near-zero constraint violations and comparable or higher performance than the baselines.Keywords
Funding Information
- NSF China (52221005, U20A20334, 52072213)
- Tsinghua University
- Toyota Motor Corporation
- Tsinghua University
- DIDI
This publication has 20 references indexed in Scilit:
- Safe-Control-Gym: A Unified Benchmark Suite for Safe Learning-Based Control and Reinforcement Learning in RoboticsIEEE Robotics and Automation Letters, 2022
- Verification and Synthesis of Control Barrier Functions2021
- High-Order Control Barrier FunctionsIEEE Transactions on Automatic Control, 2021
- Scalable Learning of Safety Guarantees for Autonomous Systems using Hamilton-Jacobi Reachability2021
- Recovery RL: Safe Reinforcement Learning With Learned Recovery ZonesIEEE Robotics and Automation Letters, 2021
- Mastering Atari, Go, chess and shogi by planning with a learned modelNature, 2020
- Learning Control Barrier Functions from Expert Demonstrations2020
- Synthesis of Control Barrier Functions Using a Supervised Machine Learning Approach2020
- A General Safety Framework for Learning-Based Control in Uncertain Robotic SystemsIEEE Transactions on Automatic Control, 2018
- Control Barrier Function Based Quadratic Programs for Safety Critical SystemsIEEE Transactions on Automatic Control, 2016