Cost-Effective Extension of DRAM-PIM for Group-Wise LLM Quantization
Open Access
- 13 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (1) , 53-56
Abstract
Processing-in-Memory (PIM) is emerging as a promising next-generation hardware to address memory bottlenecks in large language model (LLM) inference by leveraging internal memory bandwidth, enabling more energy-efficient on-device AI. However, LLMs’ large footprint poses significant challenges for accelerating them on PIM due to limited available space. Recent advances in weight-only quantization, especially group-wise weight quantization (GWQ), reduce LLM model sizes, enabling parameters to be stored at 4-bit precision or lower with minimal accuracy loss. Despite this, current PIM architectures experience performance degradation when handling the additional computations required for quantized weights. While incorporating extra logic could mitigate this degradation, it is often prohibitively expensive due to the constraints of memory technology, necessitating solutions with minimal area overhead. This work introduces two key innovations: 1) scale cascading, and 2) an INT2FP converter, to support GWQ-applied LLMs on PIM with minimal dequantization latency and area overhead compared to FP16 GEMV. Experimental results show that the proposed approach adds less than 0.6% area overhead to the existing PIM unit and achieves a 7% latency overhead for dequantization and GEMV in 4-bit GWQ with a group size of 128, compared to FP16 GEMV, while offering a 1.55× performance gain over baseline dequantization.Keywords
Funding Information
- IITP
- Korea governmen (RS-2019-II191906)
- Samsung Advanced Institute of Technology
This publication has 8 references indexed in Scilit:
- AttAcc! Unleashing the Power of PIM for Batched Transformer-based Generative Model Inference2024
- NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing2024
- TransPIM: A Memory-based Acceleration via Software-Hardware Co-Design for Transformer2022
- Who Says Elephants Can’t Run: Bringing Large Scale MoE Models into Cloud Scale Production2022
- Hardware Architecture and Software Stack for PIM Based on Commercial DRAM Technology : Industrial Product2021
- Quant-PIM: An Energy-Efficient Processing-in-Memory Accelerator for Layerwise Quantized Neural NetworksIEEE Embedded Systems Letters, 2021
- McDRAM: Low Latency and Energy-Efficient Matrix Computations in DRAMIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2018
- DRAMSim2: A Cycle Accurate Memory System SimulatorIEEE Computer Architecture Letters, 2011