Efficient memristor accelerator for transformer self-attention functionality
Open Access
- 15 October 2024
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 14 (1) , 1-15
Abstract
The adoption of transformer networks has experienced a notable surge in various AI applications. However, the increased computational complexity, stemming primarily from the self-attention mechanism, parallels the manner in which convolution operations constrain the capabilities and speed of convolutional neural networks (CNNs). The self-attention algorithm, specifically the matrix-matrix multiplication (MatMul) operations, demands a substantial amount of memory and computational complexity, thereby restricting the overall performance of the transformer. This paper introduces an efficient hardware accelerator for the transformer network, leveraging memristor-based in-memory computing. The design targets the memory bottleneck associated with MatMul operations in the self-attention process, utilizing approximate analog computation and the highly parallel computations facilitated by the memristor crossbar architecture. Remarkably, this approach resulted in a reduction of approximately 10 times in the number of multiply-accumulate (MAC) operations in transformer networks, while maintaining 95.47% accuracy for the MNIST dataset, as validated by a comprehensive circuit simulator employing NeuroSim 3.0. Simulation outcomes indicate an area utilization of 6895.7 \(\mu m^2\), a latency of 15.52 seconds, an energy consumption of 3 mJ, and a leakage power of 59.55 \(\mu W\). The methodology outlined in this paper represents a substantial stride towards a hardware-friendly transformer architecture for edge devices, poised to achieve real-time performance.
Keywords
All Related Versions
This publication has 49 references indexed in Scilit:
- Unidirectional Neuromorphic Resistive Memory Integrated with Piezoelectric Nanogenerator for Self‐Power ElectronicsAdvanced Functional Materials, 2023
- Stopping Voltage‐Dependent PCM and RRAM‐Based Neuromorphic Characteristics of Germanium TellurideAdvanced Functional Materials, 2023
- A bioinspired flexible neuromuscular system based thermal-annealing-free perovskite with passivationNature Communications, 2022
- Asymmetric GaN/ZnO Engineered Resistive Memory Device for Electronic SynapsesACS Applied Electronic Materials, 2022
- RRAM-based CAM combined with time-domain circuits for hyperdimensional computingScientific Reports, 2021
- Down-Scalable and Ultra-fast Memristors with Ultra-high Density Three-Dimensional Arrays of Perovskite Quantum WiresNano Letters, 2021
- 90% yield production of polymer nano-memristor for in-memory computingNature Communications, 2021
- NeuroMem: Analog Graphene-Based Resistive Memory for Artificial Neural NetworksScientific Reports, 2020
- Analogue signal and image processing with large memristor crossbarsNature Electronics, 2017
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998