Optimistic Temporal Difference Learning for 2048
- 3 September 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 14 (3) , 478-487
Abstract
Temporal difference (TD) learning and its variants, such as multistage TD learning and temporal coherence (TC) learning, have been successfully applied to 2048 . These methods rely on the stochasticity of the environment of 2048 for exploration. In this article, we propose to employ optimistic initialization (OI) to encourage exploration for 2048 , and empirically show that the learning quality is significantly improved. This approach optimistically initializes the feature weights to very large values. Since weights tend to be reduced once the states are visited, agents tend to explore those states which are unvisited or visited few times. Our experiments show that both TD and TC learning with OI significantly improve the performance. As a result, the network size required to achieve the same performance is significantly reduced. With additional tunings such as expectimax search, multistage learning, and tile-downgrading technique, our design achieves the state-of-the-art performance, namely an average score of 625 377 and a rate of 72% reaching 32 768-tiles. In addition, for sufficiently large tests, 65 536-tiles are reached at a rate of 0.02%.Keywords
All Related Versions
Funding Information
- Ministry of Science and Technology of the People's Republic of China (108-2634-F-009-011, 109-2634-F-009-019, 110-2634-F-009-022)
- Pervasive Artificial Intelligence Research Laboratory
- National Yang Ming Chiao Tung University
- Ministry of Education, Taiwan (110W265)
- National Center for High-performance Computing (MST108115)
This publication has 22 references indexed in Scilit:
- Multistage Temporal Difference Learning for 2048-Like GamesIEEE Transactions on Computational Intelligence and AI in Games, 2016
- Temporal difference learning of N-tuple networks for the game 20482014
- Multi-Stage Temporal Difference Learning for 20482014
- Temporal Difference Learning for Connect6Lecture Notes in Computer Science, 2012
- Reinforcement Learning with N-tuples on the Game Connect-42012
- Recent Advances in Hierarchical Reinforcement LearningDiscrete Event Dynamic Systems, 2003
- First Results from Using Temporal Difference Learning in ShogiLecture Notes in Computer Science, 1999
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- Learning to predict by the methods of temporal differencesMachine Learning, 1988
- The *-minimax search procedure for trees containing chance nodesArtificial Intelligence, 1983