Cell-aware Stacked LSTMs for Modeling Sentences
Authors:
Preprint
Abstract
We propose a method of stacking multiple long short-term memory (LSTM) layers for modeling sentences. In contrast to the conventional stacked LSTMs where only hidden states are fed as input to the next layer, the suggested architecture accepts both hidden and memory cell states of the preceding layer and fuses information from the left and the lower context using the soft gating mechanism of LSTMs. Thus the architecture modulates the amount of information to be delivered not only in horizontal recurrence but also in vertical connections, from which useful features extracted from lower layers are effectively conveyed to upper layers. We dub this architecture Cell-aware Stacked LSTM (CAS-LSTM) and show from experiments that our models bring significant performance gain over the standard LSTMs on benchmark datasets for natural language inference, paraphrase detection, sentiment classification, and machine translation. We also conduct extensive qualitative analysis to understand the internal behavior of the suggested approach.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Do latent tree learning models identify meaningful structure in sentences?Transactions of the Association for Computational Linguistics, 2018
- On Tree-Based Neural Sentence Modeling2018
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data2017
- Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature RepresentationsTransactions of the Association for Computational Linguistics, 2016
- Highway long short-term memory RNNS for distant speech recognition2016
- Sequence-to-Sequence Learning as Beam-Search Optimization2016
- Document Modeling with Gated Recurrent Neural Network for Sentiment Classification2015
- Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank2013
- Learning task-dependent distributed representations by backpropagation through structure2002
- Learning Complex, Extended Sequences Using the Principle of History CompressionNeural Computation, 1992