AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
Preprint
Abstract
Transformer models gain popularity because of their superior inference accuracy and inference throughput. However, the transformer is computation-intensive, causing a long inference time. The existing works on transformer inference acceleration have limitations caused by either the modification of transformer architectures or the need of specialized hardware. In this paper, we identify the opportunities of using memoization to accelerate the self-attention mechanism in transformers without the above limitations. Built upon a unique observation that there is rich similarity in attention computation across inference sequences, we build a memoization database that leverages the emerging big memory system. We introduce a novel embedding technique to find semantically similar inputs to identify computation similarity. We also introduce a series of techniques such as memory mapping and selective memoization to avoid memory copy and unnecessary overhead. We enable 22% inference-latency reduction on average (up to 68%) with negligible loss in inference accuracy.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Learned Token Pruning for Transformers2022
- Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm2021
- DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering2020
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Adaptive Attention Span in Transformers2019
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Selective Convolutional Descriptor Aggregation for Fine-Grained Image RetrievalIEEE Transactions on Image Processing, 2017