Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games With Local Control and Global State Information
- 19 October 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automatic Control
- Vol. 67 (10) , 5230-5245
Abstract
Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multiagent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this article, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are “independent learners”: they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to the best of our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games.Keywords
All Related Versions
Funding Information
- Natural Sciences and Engineering Research Council of Canada
This publication has 39 references indexed in Scilit:
- Learning efficient Nash equilibria in distributed systemsGames and Economic Behavior, 2012
- Revisiting log-linear learning: Asynchrony, completeness and payoff-based implementationGames and Economic Behavior, 2012
- Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problemsThe Knowledge Engineering Review, 2012
- Infinite horizon common interest games with perfect informationGames and Economic Behavior, 2005
- Multiagent learning using a variable learning rateArtificial Intelligence, 2002
- Value-function reinforcement learning in Markov gamesCognitive Systems Research, 2001
- Stochastic approximation with two time scalesSystems & Control Letters, 1997
- Non zero-sum stochastic games in admission, service and routing control in queueing systemsQueueing Systems, 1996
- Asynchronous stochastic approximation and Q-learningMachine Learning, 1994
- The Evolution of ConventionsEconometrica, 1993