Deterministic Sequencing of Exploration and Exploitation for Reinforcement Learning
Preprint
Abstract
We propose Deterministic Sequencing of Exploration and Exploitation (DSEE) algorithm with interleaving exploration and exploitation epochs for model-based RL problems that aim to simultaneously learn the system model, i.e., a Markov decision process (MDP), and the associated optimal policy. During exploration, DSEE explores the environment and updates the estimates for expected reward and transition probabilities. During exploitation, the latest estimates of the expected reward and transition probabilities are used to obtain a robust policy with high probability. We design the lengths of the exploration and exploitation epochs such that the cumulative regret grows as a sub-linear function of time.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Structural properties of optimal fidelity selection policies for human-in-the-loop queuesAutomatica, 2023
- On Regret-Optimal Learning in Decentralized Multiplayer Multiarmed BanditsIEEE Transactions on Control of Network Systems, 2016
- Automatic web content personalization through reinforcement learningJournal of Systems and Software, 2016
- Deterministic Sequencing of Exploration and Exploitation for Multi-Armed Bandit ProblemsIEEE Journal of Selected Topics in Signal Processing, 2013
- Learning in a Changing World: Restless Multiarmed Bandit With Unknown DynamicsIEEE Transactions on Information Theory, 2012
- Robust Adaptive Markov Decision Processes: Planning with Model UncertaintyIEEE Control Systems, 2012
- An analysis of model-based Interval Estimation for Markov Decision ProcessesJournal of Computer and System Sciences, 2008
- The expected hitting times for finite Markov chainsLinear Algebra and its Applications, 2008
- Robust Control of Markov Decision Processes with Uncertain Transition MatricesOperations Research, 2005
- Robust Dynamic ProgrammingMathematics of Operations Research, 2005