Applying Quantitative Model Checking to Analyze Safety in Reinforcement Learning
Open Access
- 25 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 18957-18971
Abstract
Reinforcement learning (RL) is rapidly used in safety-centric applications. However, many studies focus on generating optimal policy that achieves maximum rewards. While maximum rewards are beneficial, safety constraints and non-functional requirements must also be considered in safety-centric applications to avoid dangerous situations. For example, in the case of food delivery robots in restaurants, RL should be used not only to find optimal policy that response to all customer requests through maximum rewards but also to consider safety constraints such as collision avoidance and non-functional requirements such as battery saving. In this paper, we investigated the fulfillment of safety constraints and non-functional requirements of learning models generated through RL with quantitative model checking. We experimented with various time steps and learning rates required for RL, targeting restaurant delivery robots. The functional requirement of these robots is to process all customer order requests, and the non-functional requirements are the number of steps and battery consumption to complete the task. Safety constraints include the amount of collision and the probability of collision. Through these experiments, we made three important findings. First, learning models that obtain maximum rewards may have a low degree of achievement of non-functional requirements and safety constraints. Second, as safety constraints are met, the degree of achievement of non-functional requirements may be low. Third, even if the maximum reward is not obtained, sacrificing non-functional requirements can maximize the achievement of safety constraints. These results show that learning models generated through RL can trade off rewards to achieve safety constraints. In conclusion, our work can contribute to selecting suitable hyperparameters and optimal learning models during RL.Keywords
Funding Information
- Institute of Information & Communications Technology Planning & Evaluation (2021-0-00122)
This publication has 18 references indexed in Scilit:
- Mobile agent path planning under uncertain environment using reinforcement learning and probabilistic model checkingKnowledge-Based Systems, 2023
- Safe reinforcement learning under temporal logic with reward design and quantum action selectionScientific Reports, 2023
- Dependability Analysis of Deep Reinforcement Learning based Robotics and Autonomous Systems through Probabilistic Model Checking2022
- Temporal logic guided safe model-based reinforcement learning: A hybrid systems approachNonlinear Analysis: Hybrid Systems, 2022
- Machine Learning: Algorithms, Real-World Applications and Research DirectionsSN Computer Science, 2021
- Synthesizing safe policies under probabilistic constraints with reinforcement learning and Bayesian model checkingScience of Computer Programming, 2021
- Statistically Model Checking PCTL Specifications on Markov Decision Processes via Reinforcement Learning2020
- Appraising research on personalized learning: Definitions, theoretical alignment, advancements, and future directionsJournal of Research on Technology in Education, 2020
- On Probabilistic Computation Tree Logic2004
- PRISM: Probabilistic Symbolic Model Checker2002