ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads
Top Cited Papers
Open Access
- 30 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Architecture and Code Optimization
- Vol. 22 (2) , 1-24
Abstract
Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in ShuffleInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that ShuffleInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in terms of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.Keywords
Funding Information
- Strategic Priority Research Program of Chinese Academy of Sciences (XDA0320000 and XDA0320300)
- National Natural Science Foundation of China (62090022, U24B6012 and 62172388)
- China Postdoctoral Science Foundation (2024M762550)
- Shaanxi Postdoctoral Research Foundation (2024BSHSDZZ102)
This publication has 10 references indexed in Scilit:
- CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving2024
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting2024
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Towards Next-Generation Intelligent Assistants Leveraging LLM Techniques2023
- ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs2023
- Clio: a hardware-software co-designed disaggregated memory system2022
- A Case for Sampling-Based Learning Techniques in Coflow SchedulingIEEE/ACM Transactions on Networking, 2022
- LightSeq: A High Performance Inference Library for Transformers2021
- PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units2020
- Efficient coflow scheduling with Varys2014