Accessing Higher-level Representations in Sequential Transformers with Feedback Memory
Authors:
- Angela FanAFAngela Fan
- Thibaut LavrilTLThibaut Lavril
- Edouard GraveEGEdouard Grave
- Armand JoulinAJArmand Joulin
- Sainbayar SukhbaatarSSSainbayar Sukhbaatar
Preprint
Abstract
Transformers are feedforward networks that can process input tokens in parallel. While this parallelization makes them computationally efficient, it restricts the model from fully exploiting the sequential nature of the input - the representation at a given layer can only access representations from lower layers, rather than the higher level representations already built in previous time steps. In this work, we propose the Feedback Transformer architecture that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. We demonstrate on a variety of benchmarks in language modeling, neural machine translation, summarization, and reinforcement learning that the increased representation capacity can improve over Transformer baselines.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Adaptive Attention Span in Transformers2019
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling2019
- Pre-trained language model representations for language generation2019
- The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation2018
- Self-Attention with Relative Position Representations2018
- Get To The Point: Summarization with Pointer-Generator Networks2017
- Neural Machine Translation of Rare Words with Subword Units2016
- Spatial localization does not require the presence of local cuesLearning and Motivation, 1981