UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads
Open Access
- 29 May 2026
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Measurement and Analysis of Computing Systems
- Vol. 10 (2) , 1-27
Abstract
Prefix caching is a key performance optimization in Large Language Model (LLM) serving systems, enabling reuse of attention Key-Value (KV) states across requests with shared prompt prefixes. However, the size of GPU memory limits cache capacity, making the eviction policy a critical factor in overall system performance. Existing systems primarily rely on simple heuristics, such as LRU, and apply the same policy across task categories, implicitly assuming homogeneous workloads. In practice, however, modern LLMs serve heterogeneous workloads that mix multi-turn conversational traffic with diverse single-turn API requests, leading to fundamentally different prefix reuse patterns. In this work, we first design a trace-driven prefix cache simulator built on vLLM to systematically characterize prefix reuse across representative workloads. Our analysis reveals two dominant reuse patterns---session reuse and structural reuse---that vary significantly across task types. Motivated by these observations, we propose UniCache, a unified eviction policy that jointly captures both reuse patterns and dynamically balances cache allocation across tasks. When implemented in vLLM, UniCache achieves substantial improvements under heterogeneous workloads, improving prefix cache hit rates by up to 17.32% and reducing inference latency by up to 3.63× compared to existing policies.Keywords
This publication has 7 references indexed in Scilit:
- Jenga: Effective Memory Management for Serving LLM with Heterogeneity2025
- CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion2025
- Stateful Large Language Model Serving with Pensieve2025
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Question Answering with Long Multiple-Span Answers2020
- The log-dynamic brain: how skewed distributions affect network operationsNature Reviews Neuroscience, 2014
- A study of replacement algorithms for a virtual-storage computerIBM Systems Journal, 1966