Ari Arapostathis

We study Markov decision processes with Polish state and action spaces. The action space is state dependent and is not necessarily compact. We first establish the existence of an optimal ergodic occupation measure using only a near-monotone hypothesis on the running cost. Then we study the well-posedness of Bellman equation, or what is commonly known as the average cost optimality equation, under the additional hypothesis of the existence of a small set. We deviate from the usual approach which is based on the vanishing discount method and instead map the problem to an equivalent one for a controlled split chain. We employ a stochastic representation of the Poisson equation to derive the Bellman equation. Next, under suitable assumptions, we establish convergence results for the 'relative value iteration' algorithm which computes the solution of the Bellman equation recursively. In addition, we present some results concerning the stability and asymptotic optimality of the associated rolling horizon policies.

Large-scale stochastic networks are widely used for a variety of systems including telecommunications, patient flows, service and data centers, and so on. Because of their complexity, ensuring the stability of these networks by allocating the required resources needed by each customer class is quite challenging. When there are sufficient resources to serve each customer class, the existence of a policy that stabilizes the system is trivial. One can decouple the network and assign the required resources to each customer class. Conversely, one can anticipate that, when each class is resourceless, it is impossible to stabilize the system independent of the policy used. However, previous analyses did not tackle this question. In this work, we assume that some classes have excess resources, whereas others are deficient. We provide a full characterization of the stability of these networks by identifying a parameter that describes the overall excess or lack of resources in the network.

We study the generalized eigenvalue problem on the whole space for a class of integro-differential elliptic operators. The nonlocal operator is over a finite measure, but this has no particular structure. Some of our results even hold for singular kernels. The first part of the paper presents results concerning the existence of a principal eigenfunction. Then we present various necessary and/or sufficient conditions for the maximum principle to hold, and use these to characterize the simplicity of the principal eigenvalue.

The convex analytic method has proved to be a very versatile method for the study of infinite horizon average cost optimal stochastic control problems. In this paper, we revisit the convex analytic method and make three primary contributions: (i) We present an existence result for controlled Markov models that lack weak continuity of the transition kernel but are strongly continuous in the action variable for every fixed state variable. (ii) For average cost stochastic control problems in standard Borel spaces, while existing results establish the optimality of stationary (possibly randomized) policies, few results are available on the optimality of deterministic policies. We review existing results and present further conditions under which an average cost optimal stochastic control problem admits optimal solutions that are deterministic stationary. (iii) We establish conditions under which the performance under stationary deterministic (and also quantized) policies is dense in the set of performance values under randomized stationary policies.

In this article, relying on Foster-Lyapunov drift conditions, we establish subexponential upper and lower bounds on the rate of convergence in the $\mathrm{L}^p$-Wasserstein distance for a class of irreducible and aperiodic Markov processes. We further discuss these results in the context of Markov L\'evy-type processes. In the lack of irreducibility and/or aperiodicity properties, we obtain exponential ergodicity in the $\mathrm{L}^p$-Wasserstein distance for a class of It\^{o} processes under an asymptotic flatness (uniform dissipativity) assumption. Lastly, applications of these results to specific processes are presented, including Langevin tempered diffusion processes, piecewise Ornstein-Uhlenbeck processes with jumps under constant and stationary Markov controls, and backward recurrence time chains, for which we provide a sharp characterization of the rate of convergence via matching upper and lower bounds.