PQCache: Product Quantization-based KVCache for Long Context LLM Inference
Top Cited Papers
Open Access
- 17 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 3 (3) , 1-30
Abstract
As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations of tokens within LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques prevalent in the data management community, we consider the storage and retrieval of KVCache as a typical embedding retrieval problem. We propose PQCache , which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, we use PQ codes and centroids to approximately identify important preceding tokens, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments demonstrate that PQCache achieves both effectiveness and efficiency, with 4.60% score improvement over existing methods on InfiniteBench and low system latency in both prefilling and decoding.Keywords
All Related Versions
This publication has 44 references indexed in Scilit:
- Deep learning-based software engineering: progress, challenges, and opportunitiesScience China Information Sciences, 2024
- A survey on large language models for recommendationWorld Wide Web, 2024
- Deep learning for code generation: a surveyScience China Information Sciences, 2024
- A Communication Theory Perspective on Prompting Engineering Methods for Large Language ModelsJournal of Computer Science and Technology, 2024
- Advances of Pipeline Model Parallelism for Deep Learning Training: An OverviewJournal of Computer Science and Technology, 2024
- DB-GPT: Large Language Model Meets DatabaseData Science and Engineering, 2024
- Large language models in health care: Development, applications, and challengesHealth Care Science, 2023
- Hetu: a highly efficient automatic parallel distributed deep learning systemScience China Information Sciences, 2022
- Survey: Transformer based video-language pre-trainingAI Open, 2022
- Convergence of Edge Computing and Deep Learning: A Comprehensive SurveyIEEE Communications Surveys & Tutorials, 2020