On-policy concurrent reinforcement learning
- 1 October 2004
- journal article
- research article
- Published by Taylor & Francis in Journal of Experimental & Theoretical Artificial Intelligence
- Vol. 16 (4) , 245-260
Abstract
When an agent learns in a multi-agent environment, the payoff it receives is dependent on the behaviour of the other agents. If the other agents are also learning, its reward distribution becomes non-stationary. This makes learning in multi-agent systems more difficult than single-agent learning. Prior attempts at value-function based learning in such domains have used off-policy Q-learning that do not scale well as the cornerstone, with restricted success. This paper studies on-policy modifications of such algorithms, with the promise of scalability and efficiency. In particular, it is proven that these hybrid techniques are guaranteed to converge to their desired fixed points under some restrictions. It is also shown, experimentally, that the new techniques can learn (from self-play) better policies than the previous algorithms (also in self-play) during some phases of the exploration.Keywords
This publication has 8 references indexed in Scilit:
- Multiagent learning using a variable learning rateArtificial Intelligence, 2002
- Convergence Results for Single-Step On-Policy Reinforcement-Learning AlgorithmsMachine Learning, 2000
- A Unified Analysis of Value-Function-Based Reinforcement-Learning AlgorithmsNeural Computation, 1999
- Fast Online Q(λ)Machine Learning, 1998
- An analysis of temporal-difference learning with function approximationIEEE Transactions on Automatic Control, 1997
- Q-learningMachine Learning, 1992
- Two-person nonzero-sum games and quadratic programmingJournal of Mathematical Analysis and Applications, 1964
- Non-Cooperative GamesAnnals of Mathematics, 1951