Reinforcement Learning Under Probabilistic Spatio-Temporal Constraints with Time Windows
Preprint
Abstract
We propose an automata-theoretic approach for reinforcement learning (RL) under complex spatio-temporal constraints with time windows. The problem is formulated using a Markov decision process under a bounded temporal logic constraint. Different from existing RL methods that can eventually learn optimal policies satisfying such constraints, our proposed approach enforces a desired probability of constraint satisfaction throughout learning. This is achieved by translating the bounded temporal logic constraint into a total automaton and avoiding "unsafe" actions based on the available prior information regarding the transition probabilities, i.e., a pair of upper and lower bounds for each transition probability. We provide theoretical guarantees on the resulting probability of constraint satisfaction. We also provide numerical results in a scenario where a robot explores the environment to discover high-reward regions while fulfilling some periodic pick-up and delivery tasks that are encoded as temporal logic constraints.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Safe reinforcement learning under temporal logic with reward design and quantum action selectionScientific Reports, 2023
- AlwaysSafe: Reinforcement Learning without Safety Constraint Violations during Training2021
- Challenges of real-world reinforcement learning: definitions, benchmarks and analysisMachine Learning, 2021
- Safe reinforcement learning: A control barrier function optimization approachInternational Journal of Robust and Nonlinear Control, 2020
- A General Safety Framework for Learning-Based Control in Uncertain Robotic SystemsIEEE Transactions on Automatic Control, 2018
- Time window temporal logicTheoretical Computer Science, 2017
- Formal Verification With Confidence Intervals to Establish Quality of Service Properties of Software SystemsIEEE Transactions on Reliability, 2015
- Refining interval temporal logic specifications1997