LMIE-BERT: A Learnable Method for Inter-Layer Ensembles to Accelerate Inference of BERT-Style Pre-trained Models
Authors:
- 4 August 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 271-277
Abstract
Pre-trained models have brought tremendous accuracy improvements to Natural Language Processing(NLP) and Computer Vision tasks, but they suffer from slow inference speed due to the heavy model, which hinders their deployment in production. The early exit methods have been proposed to accelerate the inference speed of large pre-trained models. However, these methods will lose control of accuracy at higher speed ratios. In order to balance the trade-off between model speed and accuracy better, we propose a novel early-exit mechanism called LMIE-BERT. To achieve this, we introduce a learnable method for inter-layer ensemble strategy in the internal classifier, it trains the model to fit the information from both the previous and current layers, which enables the early exit method to get more robust results. The experimental results demonstrate that LMIE-BERT can maintain over 90% of the accuracy of the original model while achieving a 4× inference speed up in multiple tasks. Our method is ahead of other early exit methods in terms of model accuracy for the same speed ratio.Keywords
This publication has 17 references indexed in Scilit:
- Elbert: Fast Albert with Confidence-Window Based Early Exit2021
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTProceedings of the AAAI Conference on Artificial Intelligence, 2020
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- Transformers: State-of-the-Art Natural Language Processing2020
- DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference2020
- Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning2020
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices2020
- FastBERT: a Self-distilling BERT with Adaptive Inference Time2020
- Q8BERT: Quantized 8Bit BERT2019
- BranchyNet: Fast inference via early exiting from deep neural networks2016