FastText.zip: Compressing text classification models
Preprint
Abstract
We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory. After considering different solutions inspired by the hashing literature, we propose a method built upon product quantization to store word embeddings. While the original technique leads to a loss in accuracy, we adapt this method to circumvent quantization artefacts. Our experiments carried out on several benchmarks show that our approach typically requires two orders of magnitude less memory than fastText while being only slightly inferior with respect to accuracy. As a result, it outperforms the state of the art by a good margin in terms of the compromise between memory usage and accuracy.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Bag of Tricks for Efficient Text Classification2017
- Learning to Hash for Indexing Big Data—A SurveyProceedings of the IEEE, 2015
- Optimized Product Quantization for Approximate Nearest Neighbor Search2013
- Optimization with Sparsity-Inducing PenaltiesFoundations and Trends in Machine Learning, 2012
- High-dimensional signature compression for large-scale image classification2011
- Iterative quantization: A procrustean approach to learning binary codes2011
- Product Quantization for Nearest Neighbor SearchIEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
- Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search2008
- A threshold of ln n for approximating set coverJournal of the ACM, 1998
- Text categorization with Support Vector Machines: Learning with many relevant features1998