Medusa: Accelerating Serverless LLM Inference with Materialization
Open Access
- 30 March 2025
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 653-668
Abstract
Serverless is a promising paradigm to provide scalable, cost-efficient, and easy-to-use model inference services. However, the cold start of model inference functions requires loading models to the devices, which incurs high latencies and undermines the benefits of serverless computing. In LLMs, things get even worse since two extra stages are introduced: a KV cache initialization stage that profiles and anticipates memory reservation for KV cache, and a capturing stage which dynamically constructs CUDA graphs for different batch sizes. Both stages are paramount to the inference performance, but become the main culprit of cold start latency. This paper proposes Medusa to mitigate the long cold start latency through state materialization. Instead of dynamic profiling and construction in the runtime, Medusa materializes the CUDA graphs as well as the information needed by the KV cache initialization in the offline phase, and restores them efficiently in the online phase. Medusa further introduces two novel techniques -- offline-online cooperated parameters restoration and triggering-kernels enhanced kernel address restoration -- to tackle non-deterministic issues in CUDA graphs. Medusa successfully materializes and restores CUDA graphs across 10 models (with a total of 139364 CUDA graph nodes), and reduces the latency of model loading by 42.5%. Under real-world LLM inference workloads, Medusa reduces the tail latency of the time to first token (TTFT) by 53.0%.Keywords
Funding Information
- National Natural Science Foundation of China (No. 62332011)
This publication has 21 references indexed in Scilit:
- FaaSnap2022
- INFless: a native serverless system for low-latency, high-throughput inference2022
- Gillis: Serving Large Neural Networks in Serverless Functions with Automatic Model Partitioning2021
- Benchmarking, analysis, and optimization of serverless function snapshots2021
- FaasCache: keeping serverless computing alive with greedy-dual caching2021
- BATCH: Machine Learning Inference Serving on Serverless Platforms with Adaptive Batching2020
- SEUSS2020
- Catalyzer2020
- Replayable Execution Optimized for Page Sharing for a Managed Runtime Environment2019
- Pipsqueak: Lean Lambdas with Large Libraries2017