On Exact Embedding Framework for Optimal Control of Markov Decision Processes
- 3 August 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automatic Control
- Vol. 69 (2) , 1316-1323
Abstract
This article deals with the embedding framework of Markov decision processes (MDPs) with discrete state and action space to find optimal actions. The optimal control problem of MDPs can be efficiently tackled by restructuring the same into an equivalent linearly-solvable Markov decision processes (LMDPs) through the method called embedding. However, state costs under the embedding may not exactly match the original costs and even assume unrealistic values. In this work, we derive a constructive sufficient condition to devise an exact embedding solution rendering the embedded state cost to match the original system. Furthermore, since, in this case, the embedding implies a transition from the discrete to continuous action space, the correlation between the obtained continuous action and an equivalent desired discrete action is investigated using a maximum a posteriori probability-based method. Finally, some examples, including mammalian cell-cycle network, are presented to demonstrate the effectiveness of the proposed method.Keywords
This publication has 20 references indexed in Scilit:
- Sampled-data Control of Probabilistic Boolean Control Networks: A Deep Reinforcement Learning ApproachInformation Sciences, 2022
- Self-triggered control of probabilistic Boolean control networks: A reinforcement learning approachJournal of the Franklin Institute, 2022
- Optimal control of probabilistic Boolean control networks: A scalable infinite horizon approachInternational Journal of Robust and Nonlinear Control, 2021
- Online Reinforcement Learning of Optimal Threshold Policies for Markov Decision ProcessesIEEE Transactions on Automatic Control, 2021
- Linearly Solvable Mean-Field Traffic Routing GamesIEEE Transactions on Automatic Control, 2020
- Learning Optimal Scheduling Policy for Remote State Estimation Under Uncertain Channel ConditionIEEE Transactions on Control of Network Systems, 2019
- Optimal Attack Energy Allocation against Remote State EstimationIEEE Transactions on Automatic Control, 2017
- Online Markov Decision Processes With Kullback–Leibler Control CostIEEE Transactions on Automatic Control, 2014
- Efficient computation of optimal actionsProceedings of the National Academy of Sciences, 2009
- Dynamical analysis of a generic Boolean model for the control of the mammalian cell cycleBioinformatics, 2006