Preprint
Abstract
We explore the problem of imitation learning (IL) in the context of mean-field games (MFGs), where the goal is to imitate the behavior of a population of agents following a Nash equilibrium policy according to some unknown payoff function. IL in MFGs presents new challenges compared to single-agent IL, particularly when both the reward function and the transition kernel depend on the population distribution. In this paper, departing from the existing literature on IL for MFGs, we introduce a new solution concept called the Nash imitation gap. Then we show that when only the reward depends on the population distribution, IL in MFGs can be reduced to single-agent IL with similar guarantees. However, when the dynamics is population-dependent, we provide a novel upper-bound that suggests IL is harder in this setting. To address this issue, we propose a new adversarial formulation where the reinforcement learning problem is replaced by a mean-field control (MFC) problem, suggesting progress in IL within MFGs may have to build upon MFC.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Safe Model-Based Multi-Agent Mean-Field Reinforcement Learning2024
- Model-free mean-field reinforcement learning: Mean-field MDP and mean-field Q-learningThe Annals of Applied Probability, 2023
- Adversarial Inverse Reinforcement Learning for Mean Field Games2023
- Q-Learning in Regularized Mean-field GamesDynamic Games and Applications, 2022
- Unified reinforcement Q-learning for mean field game and control problemsMathematics of Control, Signals, and Systems, 2022
- Applications of Mean Field Games in financial engineering and economic theory2021
- Mean-Field Controls with Q-Learning for Cooperative MARL: Convergence and Complexity AnalysisSIAM Journal on Mathematics of Data Science, 2021
- An Algorithmic Perspective on Imitation LearningFoundations and Trends in Robotics, 2018
- Modeling crowd dynamics by the mean-field limit approachMathematical and Computer Modelling, 2010