ACCL: Architecting Highly Scalable Distributed Training Systems With Highly Efficient Collective Communication Library
- 22 June 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Micro
- Vol. 41 (5) , 85-92
Abstract
Distributed systems have been widely adopted for Deep Neural Networks (DNNs) model training. However, the scalability of distributed training systems is largely bounded by the communication cost. We design a highly-efficient collective communication library, namely Alibaba Collective Communication Library(ACCL), to build distributed training systems with linear scalability. ACCL provides optimized algorithms to fully make use of heterogeneous interconnects simultaneously. And the experimental results show significant performance improvement.decoupling the algorithm and communication operations. By leveraging ACCL, we propose a hybrid algorithm to fully make use of the intra- and inter-node interconnects simultaneously. The experimental results show that the communication performance of ACCL is about 3x over NCCL. And the end-to-end performance of applications can be improved by 12.6% to 25.2%, with respect to the ratio of communication time.Keywords
This publication has 4 references indexed in Scilit:
- EFLOPS: Algorithm and System Co-Design for a High Performance Distributed Training Platform2020
- Bandwidth optimal all-reduce algorithms for clusters of workstationsJournal of Parallel and Distributed Computing, 2009
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005
- Fat-trees: Universal networks for hardware-efficient supercomputingIEEE Transactions on Computers, 1985