Mogrifier LSTM
Preprint
Abstract
Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur. Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modelling language. In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output. This mechanism affords the modelling of a richer space of interactions between inputs and their context. Equivalently, our model can be viewed as making the transition function given by the LSTM context-dependent. Experiments demonstrate markedly improved generalization on language modelling in the range of 3-4 perplexity points on Penn Treebank and Wikitext-2, and 0.01-0.05 bpc on four character-based datasets. We establish a new state of the art on all datasets with the exception of Enwik8, where we close a large gap between the LSTM and Transformer models.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
- Universal Language Model Fine-tuning for Text Classification2018
- Adversarial Example Generation with Syntactically Controlled Paraphrase Networks2018
- Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling2017
- Adversarial Examples for Evaluating Reading Comprehension Systems2017
- Assessing the Ability of LSTMs to Learn Syntax-Sensitive DependenciesTransactions of the Association for Computational Linguistics, 2016
- Analyzing the Behavior of Visual Question Answering Models2016
- Neural Machine Translation of Rare Words with Subword Units2016
- A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural NetworksAdvanced Robotics, 2008