Category-Aware Semantic Caching for Heterogeneous LLM Workloads
Preprint
Abstract
LLM serving systems process heterogeneous query workloads where different categories exhibit different characteristics. Code queries cluster densely in embedding space while conversational queries distribute sparsely. Content staleness varies from minutes (stock data) to months (code patterns). Query repetition patterns range from power-law (code) to uniform (conversation), producing long tail cache hit rate distributions: high-repetition categories achieve 40-60% hit rates while low-repetition or volatile categories achieve 5-15% hit rates. Vector databases must exclude the long tail because remote search costs (30ms) require 15--20% hit rates to break even, leaving 20-30% of production traffic uncached. Uniform cache policies compound this problem: fixed thresholds cause false positives in dense spaces and miss valid paraphrases in sparse spaces; fixed TTLs waste memory or serve stale data. This paper presents category-aware semantic caching where similarity thresholds, TTLs, and quotas vary by query category. We present a hybrid architecture separating in-memory HNSW search from external document storage, reducing miss cost from 30ms to 2ms. This reduction makes low-hit-rate categories economically viable (break-even at 3-5% versus 15-20%), enabling cache coverage across the entire workload distribution. Adaptive load-based policies extend this framework to respond to downstream model load, dynamically adjusting thresholds and TTLs to reduce traffic to overloaded models by 9-17% in theoretical projections.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Survey of vector database management systemsThe VLDB Journal, 2024
- RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale ApplicationsACM Transactions on Storage, 2021
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World GraphsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Generating Queries with Cardinality Constraints for DBMS TestingIEEE Transactions on Knowledge and Data Engineering, 2006
- A New Semantic Cache Management Method in Mobile DatabasesJournal of Computer Science, 2005
- Semantic caching of Web queriesThe VLDB Journal, 2000