Improving Throughput-oriented LLM Inference with CPU Computations
Open Access
- 13 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 233-245
Abstract
Large language models (LLMs) have recently captured the attention of a broad audience. To a large part, their exceptional performance in text generation was made possible by an exponential growth of the model parameters. This growth, however, comes at the expense of significantly higher operational costs and a decreased processing speed. Recent research has focused on running LLMs on commodity hardware, for example, by employing the memory hierarchy to augment throughput by increasing the number of batches. These studies, however, tend to overlook or inefficiently utilize the additional computational resources provided by the CPU. In this paper, we introduce a technique capable of efficiently harnessing all available computational resources through a finely tuned and dynamic workload allocation approach. This technique applies to decoder-based models on standard general-purpose hardware, effectively minimizing idle periods for both the CPU and the GPU. We conducted experiments involving various large language models, each representing distinct decoder-based architectures. Compared to the state-of-the-art, the results demonstrate a potential for an increase of up to 105% in throughput with the OPT-30B model.Keywords
Funding Information
- National Research Foundation of Korea (21A20151113068)
- Samsung Advanced Institute of Technology
- Ministry of Science and ICT, South Korea (2023-00228255)
This publication has 13 references indexed in Scilit:
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Improving Throughput-oriented Generative Inference with CPUs2023
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints2023
- DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale2022
- Dopia2022
- MSA TransformerbioRxiv, 2021
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTProceedings of the AAAI Conference on Artificial Intelligence, 2020
- BERT4Rec2019
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
- On-the-fly workload partitioning for integrated CPU/GPU architectures2018