Continuous Time Discounted Jump Markov Decision Processes: A Discrete-Event Approach
Author:
- 1 August 2004
- journal article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Mathematics of Operations Research
- Vol. 29 (3) , 492-524
Abstract
This paper introduces and develops a new approach to the theory of continuous time jump Markov decision processes (CTJMDP). This approach reduces discounted CTJMDPs to discounted semi-Markov decision processes (SMDPs) and eventually to discrete-time Markov decision processes (MDPs). The reduction is based on the equivalence of strategies that change actions between jumps and the randomized strategies that change actions only at jump epochs. This holds both for one-criterion problems and for multiple-objective problems with constraints. In particular, this paper introduces the theory for multiple-objective problems with expected total discounted rewards and constraints. If a problem is feasible, there exist three types of optimal policies: (i) nonrandomized switching stationary policies, (ii) randomized stationary policies for the CTJMDP, and (iii) randomized stationary policies for the corresponding SMDP with exponentially distributed sojourn times, and these policies can be implemented as randomized strategies in the CTJMDP.Keywords
This publication has 28 references indexed in Scilit:
- Optimal control of average reward constrained continuous-time finite Markov decision processes2004
- Nonatomic total rewards Markov decision processes with multiple criteriaJournal of Mathematical Analysis and Applications, 2002
- Convex Analytic Methods in Markov Decision Processes2002
- Constrained Discounted Semi-Markov Decision Processes2002
- Envelopes of Sets of Measures, Tightness, and Markov Control ProcessesApplied Mathematics & Optimization, 1999
- Constrained Discounted Dynamic ProgrammingMathematics of Operations Research, 1996
- Constrained Markov Decision Models with Weighted Discounted RewardsMathematics of Operations Research, 1995
- A generalization of ‘expectation equals reciprocal of intensity' to non-stationary exponential distributionsJournal of Applied Probability, 1994
- Controlled Markov Processes1979
- A Note on Memoryless Rules for Controlling Sequential Control ProcessesThe Annals of Mathematical Statistics, 1966