Neural Population Learning beyond Symmetric Zero-sum Games
Preprint
Abstract
We study computationally efficient methods for finding equilibria in n-player general-sum games, specifically ones that afford complex visuomotor skills. We show how existing methods would struggle in this setting, either computationally or in theory. We then introduce NeuPL-JPSRO, a neural population learning algorithm that benefits from transfer learning of skills and converges to a Coarse Correlated Equilibrium (CCE) of the game. We show empirical convergence in a suite of OpenSpiel games, validated rigorously by exact game solvers. We then deploy NeuPL-JPSRO to complex domains, where our approach enables adaptive coordination in a MuJoCo control domain and skill transfer in capture-the-flag. Our work shows that equilibrium convergent population learning can be implemented at scale and in generality, paving the way towards solving real-world games between heterogeneous players with mixed motives.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- dm_control: Software and tasks for continuous controlSoftware Impacts, 2020
- Grandmaster level in StarCraft II using multi-agent reinforcement learningNature, 2019
- α-Rank: Multi-Agent Evaluation by EvolutionScientific Reports, 2019
- Human-level performance in 3D multiplayer games with population-based reinforcement learningScience, 2019
- Superhuman AI for heads-up no-limit poker: Libratus beats top professionalsScience, 2018
- Mastering the game of Go without human knowledgeNature, 2017
- Finding any nontrivial coarse correlated equilibrium is hardACM SIGecom Exchanges, 2015
- The Complexity of Computing a Nash EquilibriumSIAM Journal on Computing, 2009
- Temporal difference learning and TD-GammonCommunications of the ACM, 1995
- Strategically zero-sum games: The class of games whose completely mixed equilibria cannot be improved uponInternational Journal of Game Theory, 1978