Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference
Open Access
- 4 August 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1280-1290
Abstract
Deploying pre-trained transformer models like BERT on downstream tasks in resource-constrained scenarios is challenging due to their high inference cost, which grows rapidly with input sequence length. In this work, we propose a constraint-aware and ranking-distilled token pruning method ToP, which selectively removes unnecessary tokens as input sequence passes through layers, allowing the model to improve online inference speed while preserving accuracy. ToP overcomes the limitation of inaccurate token importance ranking in the conventional self-attention mechanism through a ranking-distilled token distillation technique, which distills effective token rankings from the final layer of unpruned models to early layers of pruned models. Then, ToP introduces a coarse-to-fine pruning approach that automatically selects the optimal subset of transformer layers and optimizes token pruning decisions within these layers through improved L0 regularization. Extensive experiments on GLUE benchmark and SQuAD tasks demonstrate that ToP outperforms state-of-the-art token pruning and model compression methods with improved accuracy and speedups. ToP reduces the average FLOPs of BERT by 8.1X while achieving competitive accuracy on GLUE, and provides a real latency speedup of up to 7.4X on an Intel CPU. Code is available at https://github.com/microsoft/Moonlit/tree/main/ToPKeywords
All Related Versions
This publication has 15 references indexed in Scilit:
- SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning2021
- TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference2021
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTProceedings of the AAAI Conference on Artificial Intelligence, 2020
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- Structured Pruning of Large Language Models2020
- Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning2020
- What Does BERT Learn about the Structure of Language?2019
- The LambdaLoss Framework for Ranking Metric Optimization2018
- Know What You Don’t Know: Unanswerable Questions for SQuAD2018
- NewsWeeder: Learning to Filter Netnews1995