SLO-Aware GPU DVFS for Energy-Efficient LLM Inference Serving
Top Cited Papers
- 28 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 23 (2) , 150-153
Abstract
The increasing popularity of LLM-based chatbots combined with their reliance on power-hungry GPU infrastructure forms a critical challenge for providers: minimizing energy consumption under Service-Level Objectives (SLOs) that ensure optimal user experience. Traditional energy optimization methods fall short for LLM inference due to their autoregressive architecture, which renders them incapable of meeting a predefined SLO without energy overprovisioning . This autoregressive nature however, allows for iteration-level adjustments, enabling continuous fine-tuning of the system throughout the inference process. In this letter, we propose a solution based on iteration-level GPU Dynamic Voltage Frequency Scaling (DVFS), aiming to reduce the energy impact of LLM serving, an approach that has the potential for more than $22.8\%$ and up to $45.5\%$ energy gains when tested in real world situations under varying SLO constraints. Our approach works on top of existing LLM hosting services, requires minimal profiling and no intervention to the inference serving system.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Deep Learning Workload Scheduling in GPU Datacenters: A SurveyACM Computing Surveys, 2024
- Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service2023
- From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference2023
- Coordinated Batching and DVFS for DNN Inference on GPU AcceleratorsIEEE Transactions on Parallel and Distributed Systems, 2022
- Flex: High-Availability Datacenters With Zero Reserved Power2021
- MLPerf Inference Benchmark2020