Abstract
As large language models (LLMs) continue to scale in both size and particularly in sequence length, the softmax-attention mechanism of the underlying transformer architecture has emerged as a critical bottleneck. While in-memory computing (IMC) based architectures offer efficient acceleration for vector-matrix multiplications (VMMs), integrating softmax remains challenging due to its computational complexity. This work presents a comprehensive analysis of softmax approximations and softmax precision requirements in modern LLMs spanning model sizes from 125M to 405B parameters across several natural language processing (NLP) benchmarks. Based on these findings, we present an IMC-optimized reformulation of softmax for attention that results in negligible accuracy degradation, including (a) transformation of complexity from sequence-length-dependency to the much smaller attention head dimension and (b) merging of softmax components within the adjacent IMC’s VMM operations. We present both digital and analog implementations. Our digital approach reduces area overhead to 81.2 μm2 and energy to 0.69 pJ per input-output pair, resulting in more than 20× lower energy-delay-area product compared to state-of-the-art. Our analog implementation achieves 37.1 μm2 and 0.33 pJ by leveraging a spike-based encoding. These results highlight the importance of rethinking softmax computation for IMC to enable scalable transformer accelerators.
Funding Information
  • Ministry of Education
  • Germany under Project NEUROTEC-II (16ME0398K)
  • Germany under Project NEUROTEC-II (16ME0399)
  • Neurosys as part of the Initiative ``Cluster4Future'' funded by the BMBF (03ZU1106CB)
  • Phase II: NeuroSys as part of the Initiative ``Clusters4Future'' funded by the Federal Ministry of Research, Technology and Space (03ZU2106CB)
  • BMBF: Verbundprojekt: WestAI - AI Service Center West through BMBF (01IS22094E)