Abstract
Traditionally, DVFS has been the main mechanism to trade-off performance and power. We observe that Deep Neural Network (DNN) applications offer the possibility to trade-off performance, power, and accuracy using both DVFS and numerical precision levels. Our proposed approach, Power-Inference accuracy Trading (PIT), monitors server's load, and accordingly adjusts the precision of the DNN model and the DVFS setting of GPU to trade-off accuracy and power consumption with response time. At high loads and tight request arrivals, PIT leverages INT8-precision instructions of GPU to dynamically change the precision of deployed DNN models and boosts GPU frequency to execute requests faster at the expense of accuracy reduction and high power consumption. However, when requests' arrival rate is relaxed and there is slack time for requests, PIT deploys high precision version of models to improve accuracy and reduces GPU frequency to decrease power consumption. We implement and deploy PIT on a state-of-the-art server equipped with a Tesla P40 GPU. Experimental results demonstrate that depending on load, PIT can improve response time up to 11% compared to a job scheduler that uses only FP32 precision. It also improves the energy consumption by up to 28%, while achieving around 99.5% accuracy of sole FP32-precision.
Funding Information
  • NSF (1814920)
  • DoD ARO (W911NF-19-1-0484)