Addressing Some Limitations of Transformers with Feedback Memory
Preprint
Abstract
Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to capture temporal relations while processing input tokens in parallel. While this parallelization makes them computationally efficient, it restricts the model from fully exploiting the sequential nature of the input. The representation at a given layer can only access representations from lower layers, rather than the higher level representations already available. In this work, we propose the Feedback Transformer architecture that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. We demonstrate on a variety of benchmarks in language modeling, machine translation, and reinforcement learning that the increased representation capacity can create small, shallow models with much stronger performance than comparable Transformers.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Efficient Content-Based Sparse Attention with Routing TransformersTransactions of the Association for Computational Linguistics, 2021
- Theoretical Limitations of Self-Attention in Neural Sequence ModelsTransactions of the Association for Computational Linguistics, 2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Adaptive Attention Span in Transformers2019
- Pre-trained language model representations for language generation2019
- The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation2018
- The Importance of Being Recurrent for Modeling Hierarchical Structure2018
- Self-Attention with Relative Position Representations2018
- Get To The Point: Summarization with Pointer-Generator Networks2017
- Neural Machine Translation of Rare Words with Subword Units2016