Enforcing safety for vision-based controllers via Control Barrier Functions and Neural Radiance Fields
Preprint
Abstract
To navigate complex environments, robots must increasingly use high-dimensional visual feedback (e.g. images) for control. However, relying on high-dimensional image data to make control decisions raises important questions; particularly, how might we prove the safety of a visual-feedback controller? Control barrier functions (CBFs) are powerful tools for certifying the safety of feedback controllers in the state-feedback setting, but CBFs have traditionally been poorly-suited to visual feedback control due to the need to predict future observations in order to evaluate the barrier function. In this work, we solve this issue by leveraging recent advances in neural radiance fields (NeRFs), which learn implicit representations of 3D scenes and can render images from previously-unseen camera perspectives, to provide single-step visual foresight for a CBF-based controller. This novel combination is able to filter out unsafe actions and intervene to preserve safety. We demonstrate the effect of our controller in real-time simulation experiments where it successfully prevents the robot from taking dangerous actions.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Safe Control With Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction Methods for Robotics and ControlIEEE Transactions on Robotics, 2023
- Verification of Image-Based Neural Network Controllers Using Generative ModelsJournal of Aerospace Information Systems, 2022
- Vision-Only Robot Navigation in a Neural Radiance WorldIEEE Robotics and Automation Letters, 2022
- Learning Safe, Generalizable Perception-Based Hybrid Control With CertificatesIEEE Robotics and Automation Letters, 2022
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Agile Autonomous Driving using End-to-End Deep Imitation Learning2018
- Control Barrier Function Based Quadratic Programs for Safety Critical SystemsIEEE Transactions on Automatic Control, 2016