Mark Zhao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Mark Zhao has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Christos Kozyrakis
Stanford University, Stanford, CA, USA
1 shared publication - Michael Garland
NVIDIA, Santa Clara, USA
1 shared publication - Scott Mahlke
University of Michigan
1 shared publication - Vikram Sharma Mailthody
Stanford University & NVIDIA Shanghai Jiao Tong University University of Colorado Boulder Carnegie Mellon University NVIDIA & University of Michigan NVIDIA & Stanford University
1 shared publication Yuwei An
Stanford University & NVIDIA Shanghai Jiao Tong University University of Colorado Boulder Carnegie Mellon University NVIDIA & University of Michigan NVIDIA & Stanford University
1 shared publication- Zhiqiang Xie
Stanford University & NVIDIA Shanghai Jiao Tong University University of Colorado Boulder Carnegie Mellon University NVIDIA & University of Michigan NVIDIA & Stanford University
1 shared publication
Latest Publications
- Open Access
Strata: Hierarchical Context Caching for Long Context Language Model Serving
Large Language Models (LLMs) with expanding context windows face significant performance hurdles. While caching key-value (KV) states is critical for avoiding redundant computation, the storage footprint of long-context caches quickly exceeds GPU memory capacity, forcing production systems to adopt hierarchical caching across memory hierarchies. However, transferring large cached contexts back to the GPU introduces severe performance bottlenecks: fragmented I/O from paged layouts prevents full bandwidth utilization, and existing schedulers fail to account for cache-loading delays, leaving systems loading-bound rather than compute-bound. We present Strata, a hierarchical context caching framework designed for efficient long context LLM serving. Strata introduces GPU-assisted I/O to combat KV cache fragmentation, decoupling GPU and CPU memory layouts and employs cache-aware request scheduling to balance compute with I/O latency and overlapping unavoidable stalls with complementary tasks. Built on SGLang and deployed in production, Strata achieves up to 5x lower Time-To-First-Token (TTFT) compared to vLLM + LMCache and 3.75x speedup over NVIDIA TensorRT-LLM on long-context benchmarks, without degrading short-context performance.
Open AccessStrata: Hierarchical Context Caching for Long Context Language Model Serving
Large Language Models (LLMs) with expanding context windows face significant performance hurdles. While caching key-value (KV) states is critical for avoiding redundant computation, the storage footprint of long-context caches quickly exceeds GPU memory capacity, forcing production systems to adopt hierarchical caching across memory hierarchies. However, transferring large cached contexts back to the GPU introduces severe performance bottlenecks: fragmented I/O from paged layouts prevents full bandwidth utilization, and existing schedulers fail to account for cache-loading delays, leaving systems loading-bound rather than compute-bound. We present Strata, a hierarchical context caching framework designed for efficient long context LLM serving. Strata introduces GPU-assisted I/O to combat KV cache fragmentation, decoupling GPU and CPU memory layouts and employs cache-aware request scheduling to balance compute with I/O latency and overlapping unavoidable stalls with complementary tasks. Built on SGLang and deployed in production, Strata achieves up to 5x lower Time-To-First-Token (TTFT) compared to vLLM + LMCache and 3.75x speedup over NVIDIA TensorRT-LLM on long-context benchmarks, without degrading short-context performance.