Untangling GPU Power Consumption: Job-Level Inference in Cloud Shared Settings
Open Access
- 26 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 624-640
Abstract
As the demand for AI-driven workloads increases, the energy consumption of Graphics Processing Units (GPUs) devices has come under intense scrutiny, particularly in hyperscale data centers where large numbers of accelerators are centralized and leased to diverse clients. In the context of cloud hyperscalers, GPUs power monitoring presents several challenges that vary depending on the product offered. The monitoring capabilities of physical devices may be limited or even absent for some products. However, given the substantial energy demands of GPUs, power monitoring is essential for both cloud providers and clients. Operators require tools to manage power distribution effectively, such as balancing workloads across Power Distribution Units (PDUs), while clients need visibility into power usage to optimize their workloads for energy efficiency. To address these challenges, we propose methods for estimating the energy consumption of jobs running on GPU devices in cloud environments, spanning from shared and managed offerings like ML-as-a-Service (MLaaS) to less managed products (e.g., Infrastructure-as-a-Service (IaaS)). Our models demonstrate the benefits of sharing GPUs for small AI workloads, as well as the current sub-optimal utilization of GPUs in cloud hyperscalers, based on insights from an IaaS GPU cluster.Keywords
Funding Information
- Mitacs (IT42864)
- Inria (FrugalCloud)
This publication has 6 references indexed in Scilit:
- PowerAPI: A Python framework for building software-defined power metersJournal of Open Source Software, 2024
- GPU-NEST: Characterizing Energy Efficiency of Multi-GPU Inference ServersIEEE Computer Architecture Letters, 2020
- The Datacenter as a Computer2019
- An Automated Baseline Correction Method Based on Iterative Morphological OperationsApplied Spectroscopy, 2018
- GPU virtualization on VMware's hosted I/O architectureACM SIGOPS Operating Systems Review, 2009
- Maximum Likelihood from Incomplete Data Via the EM AlgorithmJournal of the Royal Statistical Society Series B: Statistical Methodology, 1977