Safe Deep Policy Adaptation
Authors:
- 13 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 17286-17292
Abstract
A critical goal of autonomy and artificial intelligence is enabling autonomous robots to rapidly adapt in dynamic and uncertain environments. Classic adaptive control and safe control provide stability and safety guarantees but are limited to specific system classes. In contrast, policy adaptation based on reinforcement learning (RL) offers versatility and generalizability but presents safety and robustness challenges. We propose SafeDPA, a novel RL and control framework that simultaneously tackles the problems of policy adaptation and safe reinforcement learning. SafeDPA jointly learns adaptive policy and dynamics models in simulation, predicts environment configurations, and fine-tunes dynamics models with few-shot real-world data. A safety filter based on the Control Barrier Function (CBF) on top of the RL policy is introduced to ensure safety during real-world deployment. We provide theoretical safety guarantees of SafeDPA and show the robustness of SafeDPA against learning errors and extra perturbations. Comprehensive experiments on (1) classic control problems (Inverted Pendulum), (2) simulation benchmarks (Safety Gym), and (3) a real-world agile robotics platform (RC Car) demonstrate great superiority of SafeDPA in both safety and task performance, over state-of-the-art baselines. Particularly, SafeDPA demonstrates notable generalizability, achieving a 300% increase in safety rate compared to the baselines, under unseen disturbances in real-world experiments.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Performance, Precision, and Payloads: Adaptive Nonlinear MPC for QuadrotorsIEEE Robotics and Automation Letters, 2021
- Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey2020
- Control Barrier Function Based Quadratic Programs for Safety Critical SystemsIEEE Transactions on Automatic Control, 2016
- Control barrier function based quadratic programs with application to adaptive cruise control2014
- Reactive Sliding-Mode Algorithm for Collision Avoidance in Robotic SystemsIEEE Transactions on Control Systems Technology, 2013
- MuJoCo: A physics engine for model-based control2012
- Robust Adaptive Control2012
- Real-Time Obstacle Avoidance for Manipulators and Mobile Robots1986
- Parameter identification of robot dynamics1985
- Liapunov redesign of model reference adaptive control systemsIEEE Transactions on Automatic Control, 1966