QWQizhen Weng
Qizhen Weng
*This is an algorithmically generated author record based on publicly available information.
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Qizhen Weng has a total of 4 co-authors. Here are the top co-authors by number of shared publications:
- Baochun Li
University of Toronto
1 shared publication - Bo Li
The Hong Kong University of Science and Technology
1 shared publication - Chen Chen
Shanghai Jiao Tong University, Shanghai, China
1 shared publication - Wei Wang
HKUST, Hong Kong, Hong Kong
1 shared publication
Latest Publications
- 5 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 11 (1) , 515-531
Accelerating Distributed Learning in Non-Dedicated Environments
Machine learning (ML) models are increasingly trained with distributed workers possessing heterogeneous resources. In such scenarios, model training efficiency may be negatively affected by \emph{stragglers}---workers that run much slower than others. Efficient model training requires eliminating such stragglers, yet for modern ML workloads, existing load balancing strategies are inefficient and even infeasible. In this paper, we propose a novel strategy, called \emph{semi-dynamic load balancing}, to eliminate stragglers of distributed ML workloads. The key insight is that ML workers shall be load-balanced at \emph{iteration boundaries}, being non-intrusive to intra-iteration execution. Based on it we further develop LB-BSP, an integrated worker coordination mechanism that adapts workers' load to their instantaneous processing capabilities---by right-sizing the sample batches at the synchronization barriers. We have designed distinct load tuning algorithms for ML in CPU clusters, in GPU clusters as well as in federated learning setups, based on their respective characteristics. LB-BSP has been implemented as a Python module for ML frameworks like TensorFlow and PyTorch. Our EC2 deployment confirms that LB-BSP is practical, effective and light-weight, and is able to accelerating distributed training by up to $54\%$ .- 5 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 11 (1) , 515-531
Accelerating Distributed Learning in Non-Dedicated Environments
Machine learning (ML) models are increasingly trained with distributed workers possessing heterogeneous resources. In such scenarios, model training efficiency may be negatively affected by \emph{stragglers}---workers that run much slower than others. Efficient model training requires eliminating such stragglers, yet for modern ML workloads, existing load balancing strategies are inefficient and even infeasible. In this paper, we propose a novel strategy, called \emph{semi-dynamic load balancing}, to eliminate stragglers of distributed ML workloads. The key insight is that ML workers shall be load-balanced at \emph{iteration boundaries}, being non-intrusive to intra-iteration execution. Based on it we further develop LB-BSP, an integrated worker coordination mechanism that adapts workers' load to their instantaneous processing capabilities---by right-sizing the sample batches at the synchronization barriers. We have designed distinct load tuning algorithms for ML in CPU clusters, in GPU clusters as well as in federated learning setups, based on their respective characteristics. LB-BSP has been implemented as a Python module for ML frameworks like TensorFlow and PyTorch. Our EC2 deployment confirms that LB-BSP is practical, effective and light-weight, and is able to accelerating distributed training by up to $54\%$ .