Learned Token Pruning for Transformers
Authors:
Open Access
- 14 August 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 784-794
Abstract
Efficient deployment of transformer models in practice is challenging due to their inference cost including memory footprint, latency, and power consumption, which scales quadratically with input sequence length. To address this, we present a novel token reduction method dubbed Learned Token Pruning (LTP) which adaptively removes unimportant tokens as an input sequence passes through transformer layers. In particular, LTP prunes tokens with an attention score below a threshold, whose value is learned for each layer during training. Our threshold-based method allows the length of the pruned sequence to vary adaptively based on the input sequence, and avoids algorithmically expensive operations such as top-k token selection. We extensively test the performance of LTP on GLUE and SQuAD tasks and show that our method outperforms the prior state-of-the-art token pruning methods by up to ∽2.5% higher accuracy with the same amount of FLOPs. In particular, LTP achieves up to 2.1× FLOPs reduction with less than 1% accuracy drop, which results in up to 1.9× and 2.0× throughput improvement on Intel Haswell CPUs and NVIDIA V100 GPUs. Furthermore, we demonstrate that LTP is more robust than prior methods to variations in input sequence lengths. Our code has been developed in PyTorch and open-sourcedKeywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Efficient Content-Based Sparse Attention with Routing TransformersTransactions of the Association for Computational Linguistics, 2021
- EBERT: Efficient BERT Inference with Dynamic Structured Pruning2021
- GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference2020
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTProceedings of the AAAI Conference on Artificial Intelligence, 2020