Algorithm-Hardware Implications of Softmax Approximations for In-Memory Computing-Based LLM Accelerators
Open Access
- 6 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
- Vol. 16 (2) , 483-499
Abstract
As large language models (LLMs) continue to scale in both size and particularly in sequence length, the softmax-attention mechanism of the underlying transformer architecture has emerged as a critical bottleneck. While in-memory computing (IMC) based architectures offer efficient acceleration for vector-matrix multiplications (VMMs), integrating softmax remains challenging due to its computational complexity. This work presents a comprehensive analysis of softmax approximations and softmax precision requirements in modern LLMs spanning model sizes from 125M to 405B parameters across several natural language processing (NLP) benchmarks. Based on these findings, we present an IMC-optimized reformulation of softmax for attention that results in negligible accuracy degradation, including (a) transformation of complexity from sequence-length-dependency to the much smaller attention head dimension and (b) merging of softmax components within the adjacent IMC’s VMM operations. We present both digital and analog implementations. Our digital approach reduces area overhead to 81.2 μm2 and energy to 0.69 pJ per input-output pair, resulting in more than 20× lower energy-delay-area product compared to state-of-the-art. Our analog implementation achieves 37.1 μm2 and 0.33 pJ by leveraging a spike-based encoding. These results highlight the importance of rethinking softmax computation for IMC to enable scalable transformer accelerators.Keywords
Funding Information
- Ministry of Education
- Germany under Project NEUROTEC-II (16ME0398K)
- Germany under Project NEUROTEC-II (16ME0399)
- Neurosys as part of the Initiative ``Cluster4Future'' funded by the BMBF (03ZU1106CB)
- Phase II: NeuroSys as part of the Initiative ``Clusters4Future'' funded by the Federal Ministry of Research, Technology and Space (03ZU2106CB)
- BMBF: Verbundprojekt: WestAI - AI Service Center West through BMBF (01IS22094E)
This publication has 34 references indexed in Scilit:
- Analog in-memory computing attention mechanism for fast and energy-efficient large language modelsNature Computational Science, 2025
- Efficient scaling of large language models with mixture of experts and 3D analog in-memory computingNature Computational Science, 2025
- Mosaic: in-memory computing and routing for small-world spike-based neuromorphic systemsNature Communications, 2024
- A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inferenceNature Electronics, 2023
- A comparison of Softmax proposals2022
- Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers2021
- A pseudo-softmax function for hardware-based high speed image classificationScientific Reports, 2021
- Hardware-Aware Softmax Approximation for Deep Neural Networks2019
- Efficient Hardware Architecture of Softmax Layer in Deep Neural Network2018
- A High-Speed and Low-Complexity Architecture for Softmax Function in Deep Learning2018