Deep Transformer Q-Networks for Partially Observable Reinforcement Learning
Authors:
- Kevin EsslingerKEKevin Esslinger
- Robert PlattRPRobert Platt
- Christopher AmatoCAChristopher Amato
Preprint
Abstract
Real-world reinforcement learning tasks often involve some form of partial observability where the observations only give a partial or noisy view of the true state of the world. Such tasks typically require some form of memory, where the agent has access to multiple past observations, in order to perform well. One popular way to incorporate memory is by using a recurrent neural network to access the agent's history. However, recurrent neural networks in reinforcement learning are often fragile and difficult to train, susceptible to catastrophic forgetting and sometimes fail completely as a result. In this work, we propose Deep Transformer Q-Networks (DTQN), a novel architecture utilizing transformers and self-attention to encode an agent's history. DTQN is designed modularly, and we compare results against several modifications to our base model. Our experiments demonstrate the transformer can solve partially observable tasks faster and more stably than previous recurrent approaches.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding2020
- Attention is not Explanation2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
- Asymmetric Actor Critic for Image-Based Robot Learning2018
- Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collectionThe International Journal of Robotics Research, 2017
- ViZDoom: A Doom-based AI research platform for visual reinforcement learning2016
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014