Semi-supervised Sequence Learning
Preprint
Abstract
We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" step for a later supervised sequence learning algorithm. In other words, the parameters obtained from the unsupervised step can be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better. With pretraining, we are able to train long short term memory recurrent networks up to a few hundred timesteps, thereby achieving strong performance in many text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- LSTM: A Search Space OdysseyIEEE Transactions on Neural Networks and Learning Systems, 2016
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition2016
- Beyond short snippets: Deep networks for video classification2015
- Show and tell: A neural image caption generator2015
- Convolutional Neural Networks for Sentence Classification2014
- Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank2013
- Learning to Forget: Continual Prediction with LSTMNeural Computation, 2000
- NewsWeeder: Learning to Filter Netnews1995
- Learning representations by back-propagating errorsNature, 1986