Dynamic Programming Principles for Mean-Field Controls with Learning
Preprint
Abstract
Dynamic programming principle (DPP) is fundamental for control and optimization, including Markov decision problems (MDPs), reinforcement learning (RL), and more recently mean-field controls (MFCs). However, in the learning framework of MFCs, DPP has not been rigorously established, despite its critical importance for algorithm designs. In this paper, we first present a simple example in MFCs with learning where DPP fails with a mis-specified Q function; and then propose the correct form of Q function in an appropriate space for MFCs with learning. This particular form of Q function is different from the classical one and is called the IQ function. In the special case when the transition probability and the reward are independent of the mean-field information, it integrates the classical Q function for single-agent RL over the state-action distribution. In other words, MFCs with learning can be viewed as lifting the classical RLs by replacing the state-action space with its probability distribution space. This identification of the IQ function enables us to establish precisely the DPP in the learning framework of MFCs. Finally, we illustrate through numerical experiments the time consistency of this IQ function.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- A McKean–Vlasov approach to distributed electricity generation developmentMathematical Methods of Operations Research, 2019
- Optimal Social Policies in Mean Field GamesApplied Mathematics & Optimization, 2017
- Discrete Time McKean–Vlasov Control Problem: A Dynamic Programming ApproachApplied Mathematics & Optimization, 2016
- Dynamic programming for mean-field type controlComptes Rendus Mathématique, 2014
- Algorithmic aspects of mean–variance optimization in Markov decision processesEuropean Journal of Operational Research, 2013
- Mean Field Games and Mean Field Type Control Theory2013
- A General Stochastic Maximum Principle for SDEs of Mean-field TypeApplied Mathematics & Optimization, 2011
- A Maximum Principle for SDEs of Mean-Field TypeApplied Mathematics & Optimization, 2010
- Controlled Markov Processes and Viscosity Solutions2006
- Learning Rates for Q-LearningLecture Notes in Computer Science, 2001