UniEP: Unified Expert-Parallel MegaKernel MoE for LLM Training
Open Access
- 13 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 387-401
Abstract
As LLM training grows increasingly resource-intensive and expert parallelism (EP) becomes essential for scaling MoE models, EP optimizations are widely adopted in production frameworks like Megatron-LM. Existing solutions often rely on ad-hoc, complex kernels that lack adaptability across diverse optimization configurations and frequently neglect numerical stability, failing to meet the strict precision requirements of large-scale training. In this paper, we introduce UniEP, a novel system that unifies diverse EP optimization strategies into a cohesive abstraction. UniEP fuses the MoE communication and computation into MegaKernels, effectively transforming complex architectural tuning into a unified parameter search space for automated adaptability. Crucially, UniEP incorporates a deterministic token ordering mechanism that guarantees numerical consistency with sequential execution, even under aggressive overlap schedules. We evaluate UniEP on GPU clusters equipped with NVIDIA Hopper GPUs. Our results demonstrate that UniEP achieves 1.03 × -1.38 × speedups over state-of-the-art work, effectively mitigating communication bottlenecks while maintaining the rigorous accuracy standards required for production LLM training.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning2024
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and BeyondarXiv, 2023
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessarXiv, 2022
- FasterMoE2022
- Breaking the computation and communication abstraction barrier in distributed machine learning workloads | Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems2022
- Efficient large-scale language model training on GPU clusters using megatron-LM2021
- Synthesizing optimal collective algorithms2021
- Triton: an intermediate language and compiler for tiled neural network computations2019
- Halide2013