Online Optimization for Offline Safe Reinforcement Learning
Preprint
Abstract
We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available at https://github.com/yassineCh/O3SRL.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Safety-Aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous DrivingIEEE Robotics and Automation Letters, 2024
- Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning2024
- Safe and Efficient: A Primal-Dual Method for Offline Convex CMDPs under Partial Data Coverage2024
- A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open ProblemsIEEE Transactions on Neural Networks and Learning Systems, 2023
- Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of PessimismIEEE Transactions on Information Theory, 2022
- Supported Policy Optimization for Offline Reinforcement Learning2022
- Bootstrapped Transformer for Offline Reinforcement Learning2022
- Mildly Conservative Q-Learning for Offline Reinforcement Learning2022
- Introduction to Online Convex OptimizationFoundations and Trends® in Optimization, 2016
- The Nonstochastic Multiarmed Bandit ProblemSIAM Journal on Computing, 2002