Advances in Pre-Training Distributed Word Representations
Preprint
Abstract
Many Natural Language Processing applications nowadays rely on pre-trained word representations estimated from large text corpora such as news collections, Wikipedia and Web Crawl. In this paper, we show how to train high-quality word vector representations by using a combination of known tricks that are however rarely used together. The main result of our work is the new set of publicly available pre-trained models that outperform the current state of the art by a large margin on a number of tasks.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Bag of Tricks for Efficient Text Classification2017
- Reading Wikipedia to Answer Open-Domain Questions2017
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data2017
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Improving Distributional Similarity with Lessons Learned from Word EmbeddingsTransactions of the Association for Computational Linguistics, 2015
- Distributional Memory: A General Framework for Corpus-Based SemanticsComputational Linguistics, 2010
- Feature hashing for large scale multitask learning2009