Characterizing Energy and Performance for Distributed Training of Large Language Models

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Distributed LLM training dominates compute, power and energy consumption in modern datacenters. Parallelization strategies are tuned to to maximize training throughput under memory constraints, while GPU frequency can be adjusted to reduce power. However, these knobs are usually tuned in isolation, and their combined effect on energy efficiency is poorly understood. We evaluate how different parallelization strategies interact with GPU frequency scaling, measuring throughput, power draw, and total energy across different transformer models. We find that faster configurations generally reduce energy under fixed hardware settings, but sensitivity to reduced GPU frequency varies widely across parallelism strategies. As a result, achieving Paretooptimal performance-energy tradeoffs, requires joint optimization of parallelization strategies and GPU frequency. We provide guidelines for scheduling and modeling distributed LLM training to balance performance, power, and energy constraints.

This publication has 11 references indexed in Scilit: