POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenance
Preprint
Abstract
Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the lack of availability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), require the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of transformers and long short-term memory networks, which constitute model-free RL solutions, with a model-based/model-free hybrid approach. We apply these methods to the real-world problem of optimal maintenance planning for railway assets.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Bridging POMDPs and Bayesian decision making for robust maintenance planning under model uncertainty: An application to railway systemsReliability Engineering & System Safety, 2023
- Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learningReliability Engineering & System Safety, 2023
- Optimal inspection and maintenance planning for deteriorating structural components through dynamic Bayesian networks and Markov decision processesStructural Safety, 2022
- Value of information analysis in non-stationary stochastic decision environments: A reliability-assisted POMDP approachReliability Engineering & System Safety, 2022
- Value of structural health information in partially observable stochastic environmentsStructural Safety, 2021
- Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraintsReliability Engineering & System Safety, 2021
- Study of loaded versus unloaded measurements in railway track inspectionMeasurement, 2021
- Managing engineering systems with large state and action spaces through deep reinforcement learningReliability Engineering & System Safety, 2019
- Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part II: POMDP implementationReliability Engineering & System Safety, 2014
- Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part I: TheoryReliability Engineering & System Safety, 2014