Elbert: Fast Albert with Confidence-Window Based Early Exit
Authors:
- 6 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7713-7717
Abstract
Despite the great success in Natural Language Processing (NLP) area, large pre-trained language models like BERT are not well-suited for resource-constrained or real-time applications owing to the large number of parameters and slow inference speed. Recently, compressing and accelerating BERT have become important topics. By incorporating a parameter-sharing strategy, ALBERT greatly reduces the number of parameters while achieving competitive performance. Nevertheless, ALBERT still suffers from a long inference time. In this work, we propose the ELBERT, which significantly improves the average inference speed compared to ALBERT due to the proposed confidence-window based early exit mechanism, without introducing additional parameters or extra training overhead. Experimental results show that ELBERT achieves an adaptive inference speedup varying from 2× to 10× with negligible accuracy degradation compared to AL-BERT on various datasets. Besides, ELBERT achieves higher accuracy than existing early exit methods used for accelerating BERT under the same computation cost. Furthermore, to understand the principle of the early exit mechanism, we also visualize the decision-making process of it in ELBERT. Our code is publicly available online. 1Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities
This publication has 10 references indexed in Scilit:
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTProceedings of the AAAI Conference on Artificial Intelligence, 2020
- The Right Tool for the Job: Matching Model and Instance Complexities2020
- DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference2020
- Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning2020
- FastBERT: a Self-distilling BERT with Adaptive Inference Time2020
- Q8BERT: Quantized 8Bit BERT2019
- Patient Knowledge Distillation for BERT Model Compression2019
- Revealing the Dark Secrets of BERT2019
- A Multiscale Visualization of Attention in the Transformer Model2019
- BranchyNet: Fast inference via early exiting from deep neural networks2016