KV Cache Compression Based on Token-Level Redundancy Elimination and Bit-Level Encoding
Authors:
- 28 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Large language models (LLMs) face storage and access limitations during development, requiring extensive reads of key-value (KV) cache data for each token generation. This paper designs a hardware-friendly lossless compression and decompression method for KV cache based on Token-level Redundancy Elimination and Bit-level Encoding (TREBE), reducing the amount of KV cache data that needs to be stored. In detail, token-level redundancy is first eliminated by performing subtraction between the same layers of different tokens, followed by bit-level encoding using XOR and bit-plane techniques. TREBE achieves an average compression ratio of 2.1x for the KV cache of LLaMA2-7B and OPT-6.7B models on the WikiText2 and C4 datasets, representing a 1.9x improvement over Huffman coding. With the support of TSMC 28nm process, TREBE operates at 0.9V and 0.8GHz, achieving a throughput of 6.38GB/s. Compared to Huffman coding, energy efficiency is improved by 10x, and area efficiency is enhanced by 31.2x.Keywords
Funding Information
- Research and Development
- Nanjing University
- Nanjing University
- Nanjing University
This publication has 9 references indexed in Scilit:
- CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving2024
- Explainability for Large Language Models: A SurveyACM Transactions on Intelligent Systems and Technology, 2024
- Metaphorian: Leveraging Large Language Models to Support Extended Metaphor Creation for Science Writing2023
- The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and researchResearch in Social and Administrative Pharmacy, 2023
- A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nmIEEE Journal of Solid-State Circuits, 2023
- Energon: Toward Efficient Acceleration of Transformers Using Dynamic Sparse AttentionIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2022
- EBPC: Extended Bit-Plane Compression for Deep Neural Network Inference and Training AcceleratorsIEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2019
- Compressing Sparse Ternary Weight Convolutional Neural Networks for Efficient Hardware Acceleration2019
- A 220-900mV 179Mcode/s 36pJ/code Canonical Huffman Encoder for DEFLATE Compression in 14nm CMOS2019