Transformers are Sample-Efficient World Models
Authors:
Preprint
Abstract
Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have been designed to address this issue, with learning in the imagination of a world model being one of the most prominent approaches. However, while virtually unlimited interaction with a simulated environment sounds appealing, the world model has to be accurate over extended periods of time. Motivated by the success of Transformers in sequence modeling tasks, we introduce IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer. With the equivalent of only two hours of gameplay in the Atari 100k benchmark, IRIS achieves a mean human normalized score of 1.046, and outperforms humans on 10 out of 26 games, setting a new state of the art for methods without lookahead search. To foster future research on Transformers and world models for sample-efficient reinforcement learning, we release our code and models at https://github.com/eloialonso/iris.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Mastering Atari, Go, chess and shogi by planning with a learned modelNature, 2020
- Grandmaster level in StarCraft II using multi-agent reinforcement learningNature, 2019
- A general reinforcement learning algorithm that masters chess, shogi, and Go through self-playScience, 2018
- COMPUTING “ELO RATINGS” OF MOVE PATTERNS IN THE GAME OF GO1ICGA Journal: The Journal of the Computer Games Community, 2007
- Learning to Forget: Continual Prediction with LSTMNeural Computation, 2000
- Backpropagation Applied to Handwritten Zip Code RecognitionNeural Computation, 1989