Sparsity-Controllable Dynamic Top-p MoE for Large Foundation Model Pre-training
Authors:
Preprint
Abstract
Sparse Mixture-of-Experts (MoE) architectures effectively scale model capacity by activating only a subset of experts for each input token. However, the standard Top-k routing strategy imposes a uniform sparsity pattern that ignores the varying difficulty of tokens. While Top-p routing offers a flexible alternative, existing implementations typically rely on a fixed global probability threshold, which results in uncontrolled computational costs and sensitivity to hyperparameter selection. In this paper, we propose DTop-p MoE, a sparsity-controllable dynamic Top-p routing mechanism. To resolve the challenge of optimizing a non-differentiable threshold, we utilize a Proportional-Integral (PI) Controller that dynamically adjusts the probability threshold to align the running activated-expert sparsity with a specified target. Furthermore, we introduce a dynamic routing normalization mechanism that adapts layer-wise routing logits, allowing different layers to learn distinct expert-selection patterns while utilizing a global probability threshold. Extensive experiments on Large Language Models and Diffusion Transformers demonstrate that DTop-p consistently outperforms both Top-k and fixed-threshold Top-p baselines. Our analysis confirms that DTop-p maintains precise control over the number of activated experts while adaptively allocating resources across different tokens and layers. Furthermore, DTop-p exhibits strong scaling properties with respect to expert granularity, expert capacity, model size, and dataset size, offering a robust framework for large-scale MoE pre-training.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate2024
- XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection2024
- Harder Task Needs More Experts: Dynamic Routing in MoE Models2024
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models2024
- Learning Multi-agent Behaviors from Distributed and Streaming Demonstrations2023
- Distributed Inverse Constrained Reinforcement Learning for Multi-Agent Systems2022
- Mixture-Of-Experts with Expert Choice Routing2022
- Training Language Models to Follow Instructions with Human Feedback2022
- Are NLP Models really able to Solve Simple Math Word Problems?2021
- The LAMBADA dataset: Word prediction requiring a broad discourse context2016