Mastering 2048 With Delayed Temporal Coherence Learning, Multistage Weight Promotion, Redundant Encoding, and Carousel Shaping
Open Access
- 11 January 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 10 (1) , 3-14
Abstract
2048 is an engaging single-player nondeterministic video puzzle game, which, thanks to the simple rules and hard-tomaster gameplay, has gained massive popularity in recent years. As 2048 can be conveniently embedded into the discrete-state Markov decision processes framework, we treat it as a testbed for evaluating existing and new methods in reinforcement learning. With the aim to develop a strong 2048 playing program, we employ temporal difference learning with systematic n-tuple networks. We show that this basic method can be significantly improved with temporal coherence learning, multistage function approximator with weight promotion, carousel shaping, and redundant encoding. In addition, we demonstrate how to take advantage of the characteristics of the n-tuple network, to improve the algorithmic effectiveness of the learning process by delaying the (decayed) update and applying lock-free optimistic parallelism to effortlessly make advantage of multiple CPU cores. This way, we were able to develop the best known 2048 playing program to date, which confirms the effectiveness of the introduced methods for discrete-state Markov decision problems.Keywords
All Related Versions
Funding Information
- Polish National Science Centre (DEC-2013/09/D/ST6/03932)
This publication has 24 references indexed in Scilit:
- Online Adaptable Learning Rates for the Game Connect-4IEEE Transactions on Computational Intelligence and AI in Games, 2014
- Preference Learning for Move Prediction and Evaluation Function Approximation in OthelloIEEE Transactions on Computational Intelligence and AI in Games, 2014
- Multi-Stage Temporal Difference Learning for 20482014
- Reinforcement Learning with N-tuples on the Game Connect-42012
- Using Resource-Limited Nash Memory to Improve anOthelloEvaluation FunctionIEEE Transactions on Computational Intelligence and AI in Games, 2010
- Completely Derandomized Self-Adaptation in Evolution StrategiesEvolutionary Computation, 2001
- From Simple Features to Sophisticated Evaluation Functions1999
- Temporal difference learning and TD-GammonCommunications of the ACM, 1995
- A theory of cerebellar functionMathematical Biosciences, 1971
- GAME-PLAYING AND GAME-LEARNING AUTOMATA1966