Deep Q-Learning for Nash Equilibria: Nash-DQN
- 2 January 2022
- journal article
- research article
- Published by Taylor & Francis in Applied Mathematical Finance
- Vol. 29 (1) , 62-78
Abstract
Model-free learning for multi-agent stochastic games is an active area of research. Existing reinforcement learning algorithms, however, are often restricted to zero-sum games and are applicable only in small state-action spaces or other simplified settings. Here, we develop a new data-efficient Deep-Q-learning methodology for model-free learning of Nash equilibria for general-sum stochastic games. The algorithm uses a locally linear-quadratic expansion of the stochastic game, which leads to analytically solvable optimal actions. The expansion is parametrized by deep neural networks to give it sufficient flexibility to learn the environment without the need to experience all state-action pairs. We study symmetry properties of the algorithm stemming from label-invariant stochastic games and as a proof of concept, apply our algorithm to learning optimal trading strategies in competitive electronic markets.Keywords
All Related Versions
Funding Information
- Natural Sciences and Engineering Research Council of Canada (RGPIN-2018-05705, RGPAS-2018-522715)
This publication has 15 references indexed in Scilit:
- Sigmoid-weighted linear units for neural network function approximation in reinforcement learningNeural Networks, 2018
- Probabilistic Theory of Mean Field Games with Applications I2018
- $\epsilon$-Nash Mean Field Game Theory for Nonlinear Stochastic Dynamical Systems with Major and Minor AgentsSIAM Journal on Control and Optimization, 2013
- Large-Population LQG Games Involving a Major Player: The Nash Certainty Equivalence PrincipleSIAM Journal on Control and Optimization, 2010
- A Comprehensive Survey of Multiagent Reinforcement LearningIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2008
- Mean field gamesJapanese Journal of Mathematics, 2007
- Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principleCommunications in Information and Systems, 2006
- Fictitious play in 2×n gamesJournal of Economic Theory, 2005
- 10.1162/jmlr.2003.4.6.1039Applied Physics Letters, 2000
- Potential GamesGames and Economic Behavior, 1996