Generalized Individual Q-learning for Polymatrix Games with Partial Observations
Preprint
Abstract
This paper addresses the challenge of limited observations in non-cooperative multi-agent systems where agents can have partial access to other agents' actions. We present the generalized individual Q-learning dynamics that combine belief-based and payoff-based learning for the networked interconnections of more than two self-interested agents. This approach leverages access to opponents' actions whenever possible, demonstrably achieving a faster (guaranteed) convergence to quantal response equilibrium in multi-agent zero-sum and potential polymatrix games. Notably, the dynamics reduce to the well-studied smoothed fictitious play and individual Q-learning under full and no access to opponent actions, respectively. We further quantify the improvement in convergence rate due to observing opponents' actions through numerical simulations.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Convergence of Heterogeneous Learning Dynamics in Zero-Sum Stochastic GamesIEEE Transactions on Automatic Control, 2025
- Zero-Sum Polymatrix Games: A Generalization of MinmaxMathematics of Operations Research, 2016
- Payoff-Based Dynamics for Multiplayer Weakly Acyclic GamesSIAM Journal on Control and Optimization, 2009
- Learning in perturbed asymmetric gamesGames and Economic Behavior, 2005
- Stochastic Approximations and Differential InclusionsSIAM Journal on Control and Optimization, 2005
- Individual Q-Learning in Normal Form GamesSIAM Journal on Control and Optimization, 2005
- Experience-weighted Attraction Learning in Normal Form GamesEconometrica, 1999
- Quantal Response Equilibria for Normal Form GamesGames and Economic Behavior, 1995
- Equilibria of Polymatrix GamesManagement Science, 1972