Average optimality for continuous-time Markov decision processes in Polish spaces
Authors:
- Xianping GuoXGXianping Guo
- Ulrich RiederURUlrich Rieder
Open Access
- 1 May 2006
- journal article
- Published by Institute of Mathematical Statistics in The Annals of Applied Probability
- Vol. 16 (2) , 730-756
Abstract
This paper is devoted to studying the average optimality in continuous-time Markov decision processes with fairly general state and action spaces. The criterion to be maximized is expected average rewards. The transition rates of underlying continuous-time jump Markov processes are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. We first provide two optimality inequalities with opposed directions, and also give suitable conditions under which the existence of solutions to the two optimality inequalities is ensured. Then, from the two optimality inequalities we prove the existence of optimal (deterministic) stationary policies by using the Dynkin formula. Moreover, we present a “semimartingale characterization” of an optimal stationary policy. Finally, we use a generalized Potlach process with control to illustrate the difference between our conditions and those in the previous literature, and then further apply our results to average optimal control problems of generalized birth–death systems, upwardly skip-free processes and two queueing systems. The approach developed in this paper is slightly different from the “optimality inequality approach” widely used in the previous literature.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- Drift and monotonicity conditions for continuous-time controlled markov chains with an average criterionIEEE Transactions on Automatic Control, 2003
- Continuous-time controlled Markov chainsThe Annals of Applied Probability, 2003
- A note on optimality conditions for continuous-time Markov decision processes with average cost criterionIEEE Transactions on Automatic Control, 2001
- Computable exponential convergence rates for stochastically ordered Markov processesThe Annals of Applied Probability, 1996
- Exponential and Uniform Ergodicity of Markov ProcessesThe Annals of Probability, 1995
- Criteria for ergodicity, exponential ergodicity and strong ergodicity of Markov processesJournal of Applied Probability, 1981
- Measurable selection theorems for optimization problemsmanuscripta mathematica, 1978
- Continuous Time Control of Markov Processes on an Arbitrary State Space: Discounted RewardsThe Annals of Statistics, 1976
- On Maximal Rewards and $|varepsilon$-Optimal Policies in Continuous Time Markov Decision ChainsThe Annals of Statistics, 1974
- On the integro-differential equations of purely discontinuous Markoff processesTransactions of the American Mathematical Society, 1940