Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction
Authors:
- 7 July 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 33, 853-864
Abstract
Nowadays, large language models (LLMs) are published as a service and can be accessed by various applications via APIs, also known as language-model-as-a-service (LMaaS). Without knowing the generation length of requests, existing serving systems serve requests in a first-come, first-served (FCFS) manner with a fixed batch size, which leads to two problems that affect batch serving efficiency. First, the generation lengths of requests in a batch vary, and requests with short generation lengths must wait for requests with long generation lengths to finish during the batch serving procedure. Second, requests with longer generation lengths consume more memory during serving. Without knowing the generation lengths of batched requests, the batch size is always set small to avoid the out-of-memory (OOM) error, thus preventing the GPU from being fully utilized. In this paper, we find that a significant number of popular applications in the LMaaS scenario have a positive correlation between the generation length and the length of raw user input. Based on this observation, we propose Magnus, which can accurately predict the request generation length with the user input length, application-level, and user-level semantic features. Accordingly, Magnus can achieve high request throughput by batching requests of similar generation lengths together with adaptive batch sizes. Besides, Magnus can also schedule batches with the highest response ratio next (HRRN) policy to reduce request response time. Experiments conducted on our testbed show that Magnus improves request throughput by up to 234% and reduces response time by up to 89.7% compared to baselines.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale2022
- TCB: Accelerating Transformer Inference Services with Request Concatenation2022
- Multilingual Code Snippets Training for Program TranslationProceedings of the AAAI Conference on Artificial Intelligence, 2022
- From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model CompressionProceedings of the AAAI Conference on Artificial Intelligence, 2022
- A Comprehensive Survey of Grammatical Error CorrectionACM Transactions on Intelligent Systems and Technology, 2021
- TurboTransformers2021
- Neural Machine Translation: A ReviewJournal of Artificial Intelligence Research, 2020
- Automated program repairCommunications of the ACM, 2019
- A Survey on Research of Code Comment2019