A DNN Inference Latency-aware GPU Power Management Scheme
Authors:
- 29 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 551-554
Abstract
Graphics Processing Units (GPUs) are widely used for deep learning training as well as inference due to their high processing speed and programmability. Modern GPUs dynamically adjust the clock frequency according to their power management scheme. However, under the default scheme, the clock frequency of a GPU is only determined by utilization rate while being blind to target latency SLO, leading to unnecessary high clock frequency which causes excessive power consumption. In this paper, we propose a method to increase the energy efficiency of a GPU while satisfying latency SLO through performance scaling. It dynamically monitors the queue length of the inference engine to determine the optimal clock that can satisfy latency SLO. We implemented an efficient inference service using GPU DVFS on the existing inference engine. According to the result of experiments on inference over image classification models using three types of GPUs, all the 99th percentile latency in our method satisfied latency SLO while exhibiting better power efficiency. In particular, when processing the VGG19 model on Titan RTX, the energy consumption of the GPU is reduced by up to 49.5% compared to the default clock management when processing the same request rates.Keywords
This publication has 7 references indexed in Scilit:
- MLPerf Inference Benchmark2020
- Coordinated DVFS and Precision Control for Deep Neural NetworksIEEE Computer Architecture Letters, 2019
- The Impact of GPU DVFS on the Energy and Performance of Deep Learning2019
- Swayam2017
- A survey and measurement study of GPU DVFS on energy conservationDigital Communications and Networks, 2017
- A Survey of Methods for Analyzing and Improving GPU Energy EfficiencyACM Computing Surveys, 2014
- A measurement study of GPU DVFS on energy conservation2013