Safe Deep Policy Adaptation
Authors:
Preprint
Abstract
A critical goal of autonomy and artificial intelligence is enabling autonomous robots to rapidly adapt in dynamic and uncertain environments. Classic adaptive control and safe control provide stability and safety guarantees but are limited to specific system classes. In contrast, policy adaptation based on reinforcement learning (RL) offers versatility and generalizability but presents safety and robustness challenges. We propose SafeDPA, a novel RL and control framework that simultaneously tackles the problems of policy adaptation and safe reinforcement learning. SafeDPA jointly learns adaptive policy and dynamics models in simulation, predicts environment configurations, and fine-tunes dynamics models with few-shot real-world data. A safety filter based on the Control Barrier Function (CBF) on top of the RL policy is introduced to ensure safety during real-world deployment. We provide theoretical safety guarantees of SafeDPA and show the robustness of SafeDPA against learning errors and extra perturbations. Comprehensive experiments on (1) classic control problems (Inverted Pendulum), (2) simulation benchmarks (Safety Gym), and (3) a real-world agile robotics platform (RC Car) demonstrate great superiority of SafeDPA in both safety and task performance, over state-of-the-art baselines. Particularly, SafeDPA demonstrates notable generalizability, achieving a 300% increase in safety rate compared to the baselines, under unseen disturbances in real-world experiments.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Neural-Fly enables rapid learning for agile flight in strong windsScience Robotics, 2022
- Performance, Precision, and Payloads: Adaptive Nonlinear MPC for QuadrotorsIEEE Robotics and Automation Letters, 2021
- Learning quadrupedal locomotion over challenging terrainScience Robotics, 2020
- Learning agile and dynamic motor skills for legged robotsScience Robotics, 2019
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots2018
- Preparing for the Unknown: Learning a Universal Policy with Online System Identification2017
- Control Barrier Function Based Quadratic Programs for Safety Critical SystemsIEEE Transactions on Automatic Control, 2016
- Reactive Sliding-Mode Algorithm for Collision Avoidance in Robotic SystemsIEEE Transactions on Control Systems Technology, 2013
- Parameter identification of robot dynamics1985
- Liapunov redesign of model reference adaptive control systemsIEEE Transactions on Automatic Control, 1966