Fanxin Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Fanxin Li has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Gong Zhang
Huawei Technologies CO., Ltd, Shenzhen, China
1 shared publication - Heming Cui
The University of Hong Kong, Hong Kong, Hong Kong
1 shared publication Sen Wang
Theory Lab, 2012 Labs, Huawei Technoloies, Co. Ltd, Hong Kong, SAR, China
1 shared publication- Shixiong Zhao
Department of Computer Science, The University of Hong Kong, Hong Kong, SAR, China
1 shared publication - Xiuxian Guan
University of Hong Kong, Hong Kong, China
1 shared publication - Xusheng Chen
Huawei Technologies Co., Ltd., Shenzhen, China
1 shared publication
Latest Publications
- Open Access
- 22 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 34 (5) , 1432-1449
Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models
Training a large DNN (e.g., GPT3) efficiently on commodity clouds is challenging even with the latest 3D parallel training systems (e.g., Megatron v3.0). In particular, along the pipeline parallelism dimension, computational tasks that produce a whole DNN's gradients with multiple input batches should be concurrently activated; along the data parallelism dimension, a set of heavy-weight communications (for aggregating the accumulated outputs of computational tasks) is inevitably serialized after the pipelined tasks, undermining the training performance (e.g., in Megatron, data parallelism caused all GPUs idle for over 44% of the training time) over commodity cloud networks. To deserialize these communicational and computational tasks, we propose the AIAO scheduling (for 3D parallelism) which slices a DNN into multiple segments, so that the computational tasks processing the same DNN segment can be scheduled together, and the communicational tasks that synchronize this segment can be launched and overlapped (deserialized) with other segments’ computational tasks. We realized this idea in our Fold3D training system. Extensive evaluation shows Fold3D eliminated most of the all-GPU 44% idle time in Megatron (caused by data parallelism), leading to 25.2%–42.1% training throughput improvement compared to four notable baselines over various settings; Fold3D 's high performance scaled to many GPUs.
- 22 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 34 (5) , 1432-1449
Open AccessFold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models
Training a large DNN (e.g., GPT3) efficiently on commodity clouds is challenging even with the latest 3D parallel training systems (e.g., Megatron v3.0). In particular, along the pipeline parallelism dimension, computational tasks that produce a whole DNN's gradients with multiple input batches should be concurrently activated; along the data parallelism dimension, a set of heavy-weight communications (for aggregating the accumulated outputs of computational tasks) is inevitably serialized after the pipelined tasks, undermining the training performance (e.g., in Megatron, data parallelism caused all GPUs idle for over 44% of the training time) over commodity cloud networks. To deserialize these communicational and computational tasks, we propose the AIAO scheduling (for 3D parallelism) which slices a DNN into multiple segments, so that the computational tasks processing the same DNN segment can be scheduled together, and the communicational tasks that synchronize this segment can be launched and overlapped (deserialized) with other segments’ computational tasks. We realized this idea in our Fold3D training system. Extensive evaluation shows Fold3D eliminated most of the all-GPU 44% idle time in Megatron (caused by data parallelism), leading to 25.2%–42.1% training throughput improvement compared to four notable baselines over various settings; Fold3D 's high performance scaled to many GPUs.