Distilling Neural Networks for Greener and Faster Dependency Parsing
Preprint
Abstract
The carbon footprint of natural language processing research has been increasing in recent years due to its reliance on large and inefficient neural network implementations. Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one. We use teacher-student distillation to improve the efficiency of the Biaffine dependency parser which obtains state-of-the-art performance with respect to accuracy and parsing speed (Dozat and Manning, 2017). When distilling to 20\% of the original model's trainable parameters, we only observe an average decrease of $\sim$1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.30x (1.19x) faster than the baseline model on CPU (GPU) at inference time. We also observe a small increase in performance when compressing to 80\% for some treebanks. Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and ChallengesIEEE Signal Processing Magazine, 2018
- Distilling Knowledge for Search-based Structured Prediction2018
- Knowledge distillation for small-footprint highway networks2017
- Structured Pruning of Deep Convolutional Neural NetworksACM Journal on Emerging Technologies in Computing Systems, 2017
- Universal, unsupervised (rule-based), uncovered sentiment analysisKnowledge-Based Systems, 2017
- Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature RepresentationsTransactions of the Association for Computational Linguistics, 2016
- Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser2016
- Sentic patterns: Dependency-based rules for concept-level sentiment analysisKnowledge-Based Systems, 2014
- Enhancing the generalization ability of neural networks through controlling the hidden layersApplied Soft Computing, 2009
- A simple and effective method for removal of hidden units and weightsNeurocomputing, 1994