Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning
Authors:
- René CarmonaRCRené Carmona
- Mathieu LaurièreMLMathieu Laurière
- Zongjun TanZTZongjun Tan
Preprint
Abstract
We study infinite horizon discounted Mean Field Control (MFC) problems with common noise through the lens of Mean Field Markov Decision Processes (MFMDP). We allow the agents to use actions that are randomized not only at the individual level but also at the level of the population. This common randomization allows us to establish connections between both closed-loop and open-loop policies for MFC and Markov policies for the MFMDP. In particular, we show that there exists an optimal closed-loop policy for the original MFC. Building on this framework and the notion of state-action value function, we then propose reinforcement learning (RL) methods for such problems, by adapting existing tabular and deep RL methods to the mean-field setting. The main difficulty is the treatment of the population state, which is an input of the policy and the value function. We provide convergence guarantees for tabular algorithms based on discretizations of the simplex. Neural network based algorithms are more suitable for continuous spaces and allow us to avoid discretizing the mean field state space. Numerical examples are provided.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- A Probabilistic Approach to Classical Solutions of the Master Equation for Large Population EquilibriaMemoirs of the American Mathematical Society, 2022
- Q-Learning in Regularized Mean-field GamesDynamic Games and Applications, 2022
- Numerical Resolution of McKean-Vlasov FBSDEs Using Neural NetworksMethodology and Computing in Applied Probability, 2022
- A machine learning framework for solving high-dimensional mean field game and mean field control problemsProceedings of the National Academy of Sciences, 2020
- Random Measures, Theory and Applications2017
- Mean Field Type Control with Congestion (II): An Augmented Lagrangian MethodApplied Mathematics & Optimization, 2016
- Mean-Field-Game Model for Botnet Defense in Cyber-SecurityApplied Mathematics & Optimization, 2016
- Two Numerical Approaches to Stationary Mean-Field GamesDynamic Games and Applications, 2016
- Mean Field Games: Numerical MethodsSIAM Journal on Numerical Analysis, 2010
- Learning Rates for Q-LearningLecture Notes in Computer Science, 2001