MEMO : Accelerating Transformers with Memoization on Big Memory Systems
Preprint
Abstract
Transformers gain popularity because of their superior prediction accuracy and inference throughput. However, the transformer is computation-intensive, causing a long inference time. The existing work to accelerate transformer inferences has limitations because of the changes to transformer architectures or the need for specialized hardware. In this paper, we identify the opportunities of using memoization to accelerate the attention mechanism in transformers without the above limitation. Built upon a unique observation that there is a rich similarity in attention computation across inference sequences, we build an attention database upon the emerging big memory system. We introduce the embedding technique to find semantically similar inputs to identify computation similarity. We also introduce a series of techniques such as memory mapping and selective memoization to avoid memory copy and unnecessary overhead. We enable 21% performance improvement on average (up to 68%) with the TB-scale attention database and with ignorable loss in inference accuracy.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Learned Token Pruning for Transformers2022
- TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference2021
- DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering2020
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing2020
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Adaptive Attention Span in Transformers2019
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Selective Convolutional Descriptor Aggregation for Fine-Grained Image RetrievalIEEE Transactions on Image Processing, 2017