C-Transformer: An Energy-Efficient Homogeneous DNN-Transformer/SNN-Transformer Processor for Large Language Models
- 15 April 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 60 (10) , 3802-3815
Abstract
In this article, we propose a new language model processor named the C-Transformer to address the external memory bottleneck of language models. It consists of three key functional blocks: 1) homogeneous deep-neural-network (DNN)–Transformer/spiking-Transformer core (HDSC) with a hybrid multiplication/accumulation unit (HMAU) to enhance hardware utilization; 2) output spike speculation unit (OSSU) to increase the energy efficiency of spike-domain processing; and 3) implicit weight generation unit (IWGU) with extended sign compression (ESC) to eliminate the external memory bottleneck. The chip is fabricated in Samsung’s 28-nm 1P8M CMOS technology and operates at a supply voltage of 0.7–1.1 V with a maximum frequency of 200 MHz and supports various tasks such as language modeling, translation, and summarization. For models like generative pre-trained transformer 2 (GPT-2), multilingual text-to-text transfer transformer (mT5), text-to-text transfer transformer (T5), and FairSeq MachineTranslation (FSMT), the C-Transformer achieves $0.21{\times }$ – $0.33{\times }$ computation energy and $0.37{\times }$ – $0.41{\times }$ external memory access (EMA) energy compared to the baseline. Our chip demonstrates 13.6% lower energy consumption than the previous state-of-the-art, despite having $2.1{\times }$ more parameters. Moreover, it consumes 63.8% less energy with a similar parameter size. The C-Transformer can complete various language model tasks with <1 s latency, notably FSMT in 0.09 s and GPT-2 in 0.656 s. By combining DNN-Transformer and Spiking-Transformer architectures, the C-Transformer enhances computational energy efficiency, eliminates external memory bottlenecks, and enables language models such as GPT-2 to achieve state-of-the-art performance on mobile devices.Keywords
Funding Information
- Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government [Ministry of Science and ICT (MSIT)] (RS-2024-00399817)
- IITP under the Graduate School of Artificial Intelligence Semiconductor (IITP-2024-RS-2023-00256472)
This publication has 15 references indexed in Scilit:
- 16.1 MuITCIM: A 28nm $2.24 \mu\mathrm{J}$/Token Attention-Token-Bit Hybrid Sparse Digital CIM-Based Accelerator for Multimodal Transformers2023
- 22.9 A 12nm 18.1TFLOPs/W Sparse Transformer Processor with Entropy-Based Early Exit, Mixed-Precision Predication and Fine-Grained Power Management2023
- 16.2 A 28nm 53.8TOPS/W 8b Sparse Transformer Accelerator with In-Memory Butterfly Zero Skipper for Unstructured-Pruned NN and CIM-Based Local-Attention-Reusable Engine2023
- ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design2023
- An 82nW 0.53pJ/SOP Clock-Free Spiking Neural Network with 40µs Latency for AloT Wake-Up Functions Using Ultimate-Event-Driven Bionic Architecture and Computing-in-Memory Technique2022
- A 28nm 15.59µJ/Token Full-Digital Bitline-Transpose CIM-Based Sparse Transformer Accelerator with Pipeline/Parallel Reconfigurable Modes2022
- ReckOn: A 28nm Sub-mm2 Task-Agnostic Spiking Recurrent Neural Network Processor Enabling On-Chip Learning over Second-Long Timescales2022
- A 28nm 27.5TOPS/W Approximate-Computing-Based Transformer Processor with Asymptotic Sparsity Speculating and Out-of-Order Computing2022
- 9.3 A 40nm 4.81TFLOPS/W 8b Floating-Point Training Processor for Non-Sparse Neural Networks Using Shared Exponent Bias and 24-Way Fused Multiply-Add Tree2021
- 9.2 A 28nm 12.1TOPS/W Dual-Mode CNN Processor Using Effective-Weight-Based Convolution and Error-Compensation-Based Prediction2021