Asymmetric KV Cache Compression using State-Aware Sparsity and Quantization

Authors:
Publisher website
Google Scholar
Abstract
To enable the efficient deployment of Large Language Models (LLMs) on resource-constrained devices, recent studies have explored Key-Value (KV) Cache compression, such as quantization and sparsity-based methods. However, prior works suffer from limitations in both accuracy and memory efficiency, as they overlook the structural differences between Key and Value states and the interaction between sparsity-based compression and quantization. Moreover, previous works often underexplored the variability of generated token lengths across requests, leading to inconsistent performance. To address these limitations, we present a new asymmetric, state-aware KV Cache compression pipeline. The proposed pipeline tailors quantization granularity, sparsity-based compression type, and sparsity level to each state guided by our analysis of methodological characteristics and state-specific structural properties. Furthermore, we demonstrate that the accuracy degradation and KV Cache footprint under compression are sensitive to the ordering of different operations, highlighting the need for an order-aware design. Building on our state- and order-specific insights, we assess the influence of individual channels before applying Rotary Positional Embedding (RoPE), and define a new criterion for structured channel pruning, enabling pre-RoPE pruning to mitigate accuracy loss. We also incorporate parameter-reusing techniques and avoid extra decoding-time computation and memory overhead. Experimental results demonstrate that our approach achieves superior accuracy-memory trade-offs and consistent memory savings across sequence lengths in both quantized and non-quantized settings.
Funding Information
  • National Research Foundation of Korea (NRF) funded by Korea government (RS-2025-02264052)

This publication has 11 references indexed in Scilit: