Distances for Markov chains from sample streams
Preprint
Abstract
Bisimulation metrics are powerful tools for measuring similarities between stochastic processes, and specifically Markov chains. Recent advances have uncovered that bisimulation metrics are, in fact, optimal-transport distances, which has enabled the development of fast algorithms for computing such metrics with provable accuracy and runtime guarantees. However, these recent methods, as well as all previously known methods, assume full knowledge of the transition dynamics. This is often an impractical assumption in most real-world scenarios, where typically only sample trajectories are available. In this work, we propose a stochastic optimization method that addresses this limitation and estimates bisimulation metrics based on sample access, without requiring explicit transition models. Our approach is derived from a new linear programming (LP) formulation of bisimulation metrics, which we solve using a stochastic primal-dual optimization method. We provide theoretical guarantees on the sample complexity of the algorithm and validate its effectiveness through a series of empirical evaluations.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Computational methods for adapted optimal transportThe Annals of Applied Probability, 2024
- Mastering Atari, Go, chess and shogi by planning with a learned modelNature, 2020
- Empirical Regularized Optimal Transport: Statistical Theory and ApplicationsSIAM Journal on Mathematics of Data Science, 2020
- Causal transport plans and their Monge–Kantorovich problemsStochastic Analysis and Applications, 2018
- Causal Transport in Discrete Time and ApplicationsSIAM Journal on Optimization, 2017
- A Distance For Multistage Stochastic Optimization ModelsSIAM Journal on Optimization, 2012
- Robust Stochastic Approximation Approach to Stochastic ProgrammingSIAM Journal on Optimization, 2009
- Equivalence notions and model minimization in Markov decision processesArtificial Intelligence, 2003
- Markov Decision Processes: Discrete Stochastic Dynamic ProgrammingTechnometrics, 1995