Strata: Hierarchical Context Caching for Long Context Language Model Serving
Preprint
Abstract
Large Language Models (LLMs) with expanding context windows face significant performance hurdles. While caching key-value (KV) states is critical for avoiding redundant computation, the storage footprint of long-context caches quickly exceeds GPU memory capacity, forcing production systems to adopt hierarchical caching across memory hierarchies. However, transferring large cached contexts back to the GPU introduces severe performance bottlenecks: fragmented I/O from paged layouts prevents full bandwidth utilization, and existing schedulers fail to account for cache-loading delays, leaving systems loading-bound rather than compute-bound. We present Strata, a hierarchical context caching framework designed for efficient long context LLM serving. Strata introduces GPU-assisted I/O to combat KV cache fragmentation, decoupling GPU and CPU memory layouts and employs cache-aware request scheduling to balance compute with I/O latency and overlapping unavoidable stalls with complementary tasks. Built on SGLang and deployed in production, Strata achieves up to 5x lower Time-To-First-Token (TTFT) compared to vLLM + LMCache and 3.75x speedup over NVIDIA TensorRT-LLM on long-context benchmarks, without degrading short-context performance.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- RAGCache: Efficient Knowledge Caching for Retrieval-Augmented GenerationACM Transactions on Computer Systems, 2025
- Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management2025
- CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion2025
- Stateful Large Language Model Serving with Pensieve2025
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture2023
- Kangaroo2021
- Understanding host network stack overheads2021
- Caching with Delayed Hits2020
- The NarrativeQA Reading Comprehension ChallengeTransactions of the Association for Computational Linguistics, 2018