YTYibo Tang

Yibo Tang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Yibo Tang has a total of 10 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    AdaptiveKV: Accelerating KV Cache Offloading with a Bandwidth-Adaptive Memory Allocation Mechanism

    by Yibo Tang, Lizhou Wu, Yang Ou, Sunfeng Gao, Yanjing Wang, Zicong Wang, Xingyun Qi, Jiaqing Xu, Fangxu Lv, Liquan Xiao, et al.

    The explosive growth of key-value (KV) cache size in large language model (LLM) inference poses a key challenge to the limited HBM of GPU. Offloading KV cache to host memory has become a prevalent mitigation method. However, the limited host DDR bandwidth, especially in multi-GPU inference scenarios, often leads to offloading bottlenecks, thereby restricting inference speed. Compute express link (CXL) offers a promising alternative to expand host memory capacity and bandwidth on demand. In this article, we present a bandwidth-oriented memory allocation mechanism, named AdaptiveKV, which is self-adaptive to CXL-enabled memory pools and KV cache offloading scales for LLM inference acceleration. Our systematic profiling of CXL-HBM memory bandwidth under GPU workloads reveals that conventional memory strategies neglect dynamic memory bandwidth fluctuations and various CXL memory characteristics, leading to suboptimal memory utilization. Motivated by these insights, AdaptiveKV implements three core designs: (1) a GPU memory conch model to guide memory allocation strategies, (2) a runtime predictor to predict optimal memory allocation ratios, and (3) a dynamic interleaving strategy to allocate memory pages across available NUMA nodes. Experimental results suggest that AdaptiveKV achieves a maximum speedup of 1.90× in LLM inference throughput compared with the state-of-the-art strategies. To further explore AdaptiveKV’s applicability boundary, we also present an FPGA-based CXL memory emulator with configurable performance, revealing that a CXL-to-DDR bandwidth ratio exceeding 8% yields at least a 5% speedup in LLM inference.