Memory-Safe Hierarchical Planning for Hybrid-Parallel Transformer Training
Open Access
- 25 September 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Architecture and Code Optimization
- Vol. 23 (4) , 1-25
Abstract
Training decoder-style Transformer models at scale requires composing data, tensor, pipeline, context, and, for sparse models, expert parallelism under tight device memory limits. Without a reliable pre-profiling admission model, algebraically legal but memory-infeasible configurations can consume profiling budget without yielding useful throughput measurements. Choosing a pipeline degree also leaves a structured layer-to-stage mapping problem unresolved. This article presents Shockwave, a hierarchical planner that operates through the exposed launcher and partition interfaces of Megatron-style dense and sparse Transformer runtimes. Shockwave first builds a conservative memory-feasible region with an analytical model and backend-calibrated safety margins. It then profiles a balanced baseline and a few single-dimension anchors to learn which parallel degrees improve throughput most on the target runtime and uses that ranking to select a global strategy. Finally, it solves a mixed-integer program for stage-aware layer allocation under a fixed 1F1B pipeline schedule. Implemented as a thin control layer on top of Megatron-LM and MindSpeed, Shockwave matches or exceeds expert-curated plans, achieving geometric mean throughput gains of 1.08× over expert-tuned Megatron hybrid configurations on GPUs and 1.04× over expert MindSpeed recipes on NPUs. Relative to the evaluated automatic baselines, it improves throughput by 1.33× on GPUs and 1.10× on NPUs while reducing configuration search time by 4.59× and 5.60×.Keywords
Funding Information
- National Natural Science Foundation of China (U24A20235 and T2422007)
This publication has 28 references indexed in Scilit:
- vTrain: A Simulation Framework for Evaluating Cost-Effective and Compute-Optimal Large Language Model Training2024
- Prophet: Fine-grained Load Balancing for Parallel Training of Large-scale MoE Models2023
- Merak: An Efficient Distributed DNN Training Framework With Automated 3D Parallelism for Giant Foundation ModelsIEEE Transactions on Parallel and Distributed Systems, 2023
- MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism2023
- Efficient Pipeline Planning for Expedited Distributed DNN Training2022
- TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-ParallelismIEEE Transactions on Parallel and Distributed Systems, 2021
- DAPPLE2021
- DeepSpeed2020
- PipeDream2019
- Supporting Very Large Models using Automatic Dataflow Graph Partitioning2019