Accelerating Sparse Attention with a Reconfigurable Non-volatile Processing-In-Memory Architecture
Authors:
- 9 July 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Attention-based neural networks have shown superior performance in a wide range of tasks. Non-volatile processing-in-memory (NVPIM) architecture shows its great potential to accelerate the dense attention model. However, the unique unstructured and dynamic sparsity pattern in the sparse attention model challenges the mapping efficiency of the NVPIM architecture, as the conventional NVPIM architecture uses a vector-matrix-multiplication primitives. In this paper, we propose a NVPIM architecture to accelerate a dynamic and unstructured sparse computation in the sparse attention. We aim to improve the mapping efficiency for both SDDMM and SpMM by introducing two vector-based primitives with a reconfigurable NVPIM bank. Further, based on our reconfigurable NVPIM bank, we further propose a hybrid stationary data flow to hide the latency. Our evaluation result shows that, over previous NVPIM accelerators, our design could deliver up to 12.36× performance improvement and 3.4× energy efficiency improvement on a range of vision and language tasks.Keywords
Funding Information
- National Science Foundation
- Army Research Office
This publication has 14 references indexed in Scilit:
- ReTransformer2020
- MobiLatice2020
- Lattice: An ADC/DAC-less ReRAM-based Processing-In-Memory Architecture for Accelerating Deep Convolution Neural Networks2020
- Sparse ReRAM engine2019
- Adaptively Sparse Transformers2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- ISAACACM SIGARCH Computer Architecture News, 2016
- PRIMEACM SIGARCH Computer Architecture News, 2016
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- ImageNet: A large-scale hierarchical image database2009