AdaptiveKV: Accelerating KV Cache Offloading with a Bandwidth-Adaptive Memory Allocation Mechanism
Open Access
- 7 August 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Architecture and Code Optimization
- Vol. 23 (3) , 1-25
Abstract
The explosive growth of key-value (KV) cache size in large language model (LLM) inference poses a key challenge to the limited HBM of GPU. Offloading KV cache to host memory has become a prevalent mitigation method. However, the limited host DDR bandwidth, especially in multi-GPU inference scenarios, often leads to offloading bottlenecks, thereby restricting inference speed. Compute express link (CXL) offers a promising alternative to expand host memory capacity and bandwidth on demand. In this article, we present a bandwidth-oriented memory allocation mechanism, named AdaptiveKV, which is self-adaptive to CXL-enabled memory pools and KV cache offloading scales for LLM inference acceleration. Our systematic profiling of CXL-HBM memory bandwidth under GPU workloads reveals that conventional memory strategies neglect dynamic memory bandwidth fluctuations and various CXL memory characteristics, leading to suboptimal memory utilization. Motivated by these insights, AdaptiveKV implements three core designs: (1) a GPU memory conch model to guide memory allocation strategies, (2) a runtime predictor to predict optimal memory allocation ratios, and (3) a dynamic interleaving strategy to allocate memory pages across available NUMA nodes. Experimental results suggest that AdaptiveKV achieves a maximum speedup of 1.90× in LLM inference throughput compared with the state-of-the-art strategies. To further explore AdaptiveKV’s applicability boundary, we also present an FPGA-based CXL memory emulator with configurable performance, revealing that a CXL-to-DDR bandwidth ratio exceeding 8% yields at least a 5% speedup in LLM inference.Keywords
Funding Information
- National Natural Science Foundation of China (62304257, 62332021 and 62472058)
- National Key Research and Development Program of China (2025YFB3003102)
This publication has 21 references indexed in Scilit:
- Efficient Tensor Offloading for Large Deep-Learning Model Training based on Compute Express Link2024
- A Mess of Memory System Benchmarking, Simulation and Application Profiling2024
- NeoMem: Hardware/Software Co-Design for CXL-Native Memory Tiering2024
- Enabling Efficient Large Recommendation Model Training with Near CXL Memory Processing2024
- An LPDDR-based CXL-PNM Platform for TCO-efficient Inference of Transformer-based Large Language Models2024
- Improving Key-Value Cache Performance With Heterogeneous Memory Tiering: A Case Study of Compute-Express-Link-Based Memory ExpansionIEEE Micro, 2024
- DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale2022
- Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations2020
- Differential Evolution: A Survey of the State-of-the-ArtIEEE Transactions on Evolutionary Computation, 2010
- Random ForestsMachine Learning, 2001