Model-free mean-field reinforcement learning: Mean-field MDP and mean-field Q-learning
Authors:
Top Cited Papers
- 1 December 2023
- journal article
- research article
- Published by Institute of Mathematical Statistics in The Annals of Applied Probability
- Vol. 33 (6B) , 5334-5381
Abstract
We study infinite horizon discounted mean field control (MFC) problems with common noise through the lens of mean field Markov decision processes (MFMDP). We allow the agents to use actions that are randomized not only at the individual level but also at the level of the population. This common randomization is introduced for the purpose of exploration from a reinforcement learning (RL) paradigm. It also allows us to establish connections between both closed-loop and open-loop policies for MFC and Markov policies for the MFMDP. In particular, we show that there exists an optimal closed-loop policy for the original MFC and we prove dynamic programming principles for the state and state-action value functions. Building on this framework and the notion of state-action value function, we then propose RL methods for such problems, by adapting existing tabular and deep RL methods to the mean-field setting. The main difficulty is the treatment of the population state, which is an input of the policy and the value function. We provide convergence guarantees for the tabular Q-learning algorithm based on discretizations of the simplex. We also show that neural network based deep RL algorithms are more suitable for continuous spaces as they allow us to avoid discretizing the mean field state space. Numerical examples are provided.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Bertrand and Cournot Mean Field GamesApplied Mathematics & Optimization, 2014
- Mean Field Games: Numerical Methods for the Planning ProblemSIAM Journal on Control and Optimization, 2012
- Mean Field Games and Applications2011
- A mean field approach for optimization in discrete timeDiscrete Event Dynamic Systems, 2010
- Mean Field Games: Numerical MethodsSIAM Journal on Numerical Analysis, 2010
- Mean field gamesJapanese Journal of Mathematics, 2007
- Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principleCommunications in Information and Systems, 2006
- Deep BlueArtificial Intelligence, 2002
- 10.1162/153244303768966085Applied Physics Letters, 2000
- Q-learningMachine Learning, 1992