KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
Open Access
- 11 August 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Abstract
LLMs are widely adopted in production, pushing inference systems to their limits. Disaggregated LLM serving (e.g., PD separation and KV state disaggregation) improves scalability and cost efficiency, but it also turns KV into an explicit payload crossing network and storage boundaries, making KV a dominant end-to-end bottle-neck. Existing KV compression are typically statically configured at runtime, despite production service context varies over time in workload mix, bandwidth, and SLO/quality budgets. As a result, a fixed choice can be suboptimal or even increase latency. We present KVServe, the first service-aware and adaptive KV communication compression framework for disaggregated LLM serving: KVServe (1) unifies KV compression into a modular strategy space with new components and cross-method recomposition; (2) introduces Bayesian Profiling Engine that efficiently searches this space and distills a 3D Pareto candidate set, reducing 50x offline search overhead; and (3) deploys a Service-Aware Online Controller that combines an analytical latency model with a lightweight bandit to select profiles under constraints and correct offline-to-online mismatch. Integrated into vLLM and evaluated across datasets, models, GPUs and networks, KVServe achieves up to 9.13x JCT speedup in PD-separated serving and up to 32.8x TTFT reduction in KV-disaggregated serving.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (2032023)
- National Natural Science Foundation of China (T2125013)
- Innovation Funding of ICT, CAS (E461050)
- National Key Research and Development Program of China (2025YFB300370)
This publication has 17 references indexed in Scilit:
- HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference2025
- WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling2025
- Multiplexing Dynamic Deep Learning Workloads with SLO-awareness in GPU Clusters2025
- hZCCL: Accelerating Collective Communication with Co-Designed Homomorphic Compression2024
- LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism2024
- CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving2024
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting2024
- gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusters2024
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Accelerating Distributed Deep Learning Training with Compression Assisted Allgather and Reduce-Scatter Communication2023