FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
Top Cited Papers
Open Access
- 26 May 2023
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 1 (1) , 1-19
Abstract
With the increasing data volume, there is a trend of using large-scale pre-trained models to store the knowledge into an enormous number of model parameters. The training of these models is composed of lots of dense algebras, requiring a huge amount of hardware resources. Recently, sparsely-gated Mixture-of-Experts (MoEs) are becoming more popular and have demonstrated impressive pretraining scalability in various downstream tasks. However, such a sparse conditional computation may not be effective as expected in practical systems due to the routing imbalance and fluctuation problems. Generally, MoEs are becoming a new data analytics paradigm in the data life cycle and suffering from unique challenges at scales, complexities, and granularities never before possible. In this paper, we propose a novel DNN training framework, FlexMoE, which systematically and transparently address the inefficiency caused by dynamic dataflow. We first present an empirical analysis on the problems and opportunities of training MoE models, which motivates us to overcome the routing imbalance and fluctuation problems by a dynamic expert management and device placement mechanism. Then we introduce a novel scheduling module over the existing DNN runtime to monitor the data flow, make the scheduling plans, and dynamically adjust the model-to-hardware mapping guided by the real-time data traffic. A simple but efficient heuristic algorithm is exploited to dynamically optimize the device placement during training. We have conducted experiments on both NLP models (e.g., BERT and GPT) and vision models (e.g., Swin). And results show FlexMoE can achieve superior performance compared with existing systems on real-world workloads --- FlexMoE outperforms DeepSpeed by 1.70x on average and up to 2.10x, and outperforms FasterMoE by 1.30x on average and up to 1.45x.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Hetu: a highly efficient automatic parallel distributed deep learning systemScience China Information Sciences, 2022
- Global output feedback adaptive stabilization for systems with long uncertain input delayScience China Information Sciences, 2022
- Intent-Aware Data Visualization RecommendationData Science and Engineering, 2022
- A Comparative Study of CNN- and Transformer-Based Visual Style TransferJournal of Computer Science and Technology, 2022
- TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- WATuning: A Workload-Aware Tuning System with Attention-Based Deep Reinforcement LearningJournal of Computer Science and Technology, 2021
- Graph Learning for Combinatorial Optimization: A Survey of State-of-the-ArtData Science and Engineering, 2021
- A Workload-Adaptive Streaming Partitioner for Distributed Graph StoresData Science and Engineering, 2021
- PipeDream2019