ReTransformer
Authors:
Open Access
- 2 November 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-9
Abstract
Transformer has emerged as a popular deep neural network (DNN) model for Neural Language Processing (NLP) applications and demonstrated excellent performance in neural machine translation, entity recognition, etc. However, its scaled dot-product attention mechanism in auto-regressive decoder brings a performance bottleneck during inference. Transformer is also computationally and memory intensive and demands for a hardware acceleration solution. Although researchers have successfully applied ReRAM-based Processing-in-Memory (PIM) to accelerate convolutional neural networks (CNNs) and recurrent neural networks (RNNs), the unique computation process of the scaled dot-product attention in Transformer makes it difficult to directly apply these designs. Besides, how to handle intermediate results in Matrix-matrix Multiplication (MatMul) and how to design a pipeline at a finer granularity of Transformer remain unsolved. In this work, we propose ReTransformer - a ReRAM-based PIM architecture for Transformer acceleration. ReTransformer can not only accelerate the scaled dot-product attention of Transformer using ReRAM-based PIM but also eliminate some data dependency by avoiding writing the intermediate results using the proposed matrix decomposition technique. Moreover, we propose a new sub-matrix pipeline design for multi-head self-attention. Experimental results show that compared to GPU and Pipelayer, ReTransformer improves computing efficiency by 23.21× and 3.25×, respectively. The corresponding overall power is reduced by 1086× and 2.82×, respectively.Keywords
Funding Information
- National Science Foundation (1955246, 1910299, 1725456)
- Army Research Office (W911NF-19-2-0107)
This publication has 28 references indexed in Scilit:
- PipeLayer: A Pipelined ReRAM-Based Accelerator for Deep Learning2017
- Demonstration of Logic Operations in High-Performance RRAM Crossbar Array Fabricated by Atomic Layer Deposition TechniqueDiscover Nano, 2017
- Efficient hardware architecture of softmax layer in deep neural network2016
- ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars2016
- PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory2016
- Deep Residual Learning for Image Recognition2016
- Technological Exploration of RRAM Crossbar Array for Matrix-Vector MultiplicationJournal of Computer Science and Technology, 2016
- MAGIC—Memristor-Aided LogicIEEE Transactions on Circuits and Systems II: Express Briefs, 2014
- Memristor crossbar based hardware realization of BSB recall function2012
- Long Short-Term MemoryNeural Computation, 1997