Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-Scale MoE Models
- 2 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 37 (5) , 1121-1135
Abstract
The size of deep learning models has been increasing to enhance model quality. The linear increase in training computation budgets with model size means that training an extremely large-scale model is exceedingly time-consuming. Recently, the Mixture of Experts (MoE) has drawn significant attention as it can scale models to extra-large sizes with a near-stable computation budget. However, inefficient distributed training of large-scale MoE models hinders their broader application. Specifically, a considerable dynamic load imbalance occurs among devices during training, significantly reducing throughput. Several load-balancing works have been proposed to address the challenge. System-level solutions draw more attention for their hardware affinity and non-disruption of model convergence compared to algorithm-level ones. However, they are troubled by high communication costs and poor communication-computation overlap. To address these challenges, we propose a systematic load-balancing method, Pro-Prophet, which consists of a planner and a scheduler for efficient parallel training of large-scale MoE models. To adapt to the dynamic load imbalance, we have profiled training statistics and utilized them to design Pro-Prophet. For lower communication volume, the Pro-Prophet planner determines a series of lightweight load-balancing strategies and efficiently searches for a communication-efficient one for training based on the statistics. For sufficient overlapping of communication and computation, the Pro-Prophet scheduler schedules the data-dependent operations based on the statistics and operation characteristics, further improving the training throughput. We conduct extensive experiments in various clusters and MoE models. The results indicate that Pro-Prophet achieves up to 2.66x speedup on MoE-GPT models compared to two popular MoE frameworks, namely Deepspeed-MoE and FasterMoE. Furthermore, Pro-Prophet has demonstrated a load-balancing improvement of up to 11.01x and speedups on modern MoE models up to 1.22x compared to a representative load-balancing work, FasterMoE.Keywords
Funding Information
- National Key R&D Program of China (2022YFB4501400)
- Foundation for Innovative Research Groups of the National Natural Science Foundation of China (62421002)
- Foundation for Innovative Research Groups of the National Natural Science Foundation of China (62025208)
- Foundation for Innovative Research Groups of the National Natural Science Foundation of China (62302512)
This publication has 32 references indexed in Scilit:
- Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules2024
- Prophet: Fine-grained Load Balancing for Parallel Training of Large-scale MoE Models2023
- PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining2023
- HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training2022
- Designing Effective Sparse Expert Models2022
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms2019
- Optimization of All-to-All Communication on the Blue Gene/L Supercomputer2008
- Stabilization of Planar Collective Motion: All-to-All CommunicationIEEE Transactions on Automatic Control, 2007
- The Hierarchical Factor Algorithm for All-to-All Communication2002