Learned Token Pruning for Transformers
Authors:
- Sehoon KimSKSehoon Kim
- Sheng ShenSSSheng Shen
- David ThorsleyDTDavid Thorsley
- Amir GholamiAGAmir Gholami
- Woosuk KwonWKWoosuk Kwon
- Joseph HassounJHJoseph Hassoun
- Kurt KeutzerKKKurt Keutzer
Preprint
Abstract
Deploying transformer models in practice is challenging due to their inference cost, which scales quadratically with input sequence length. To address this, we present a novel Learned Token Pruning (LTP) method which adaptively removes unimportant tokens as an input sequence passes through transformer layers. In particular, LTP prunes tokens with an attention score below a threshold value which is learned for each layer during training. Our threshold-based method allows the length of the pruned sequence to vary adaptively based on the input sequence, and avoids algorithmically expensive operations such as top-k token selection. We extensively test the performance of LTP on GLUE tasks and show that our method outperforms the prior state-of-the-art token pruning methods by up to ~2.5% higher accuracy with the same amount of FLOPs. In particular, LTP achieves up to 2.1x FLOPs reduction with less than 1% accuracy drop, which results in up to 1.9x and 2.0x throughput improvement on Intel Haswell CPUs and NVIDIA V100 GPUs, respectively. Furthermore, we demonstrate that LTP is more robust than prior methods to variations on input sentence lengths. Our code has been developed in PyTorch and has been open-sourced.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- On the effect of dropping layers of pre-trained transformer modelsComputer Speech & Language, 2022
- A Survey of Quantization Methods for Efficient Neural Network Inference2022
- SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning2021
- Efficient Content-Based Sparse Attention with Routing TransformersTransactions of the Association for Computational Linguistics, 2021
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Masking as an Efficient Alternative to Finetuning for Pretrained Language Models2020
- When BERT Plays the Lottery, All Tickets Are Winning2020
- schuBERT: Optimizing Elements of BERT2020
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices2020
- Q8BERT: Quantized 8Bit BERT2019