TernaryBERT: Distillation-aware Ultra-low Bit BERT
Preprint
Abstract
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices. In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the full-precision model while being 14.9x smaller.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference2020
- LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression2020
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- Q8BERT: Quantized 8Bit BERT2019
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Know What You Don’t Know: Unanswerable Questions for SQuAD2018
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016