Efficient sparse collective communication and its application to accelerate distributed deep learning
Authors:
Top Cited Papers
Open Access
- 9 August 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 676-691
Abstract
Efficient collective communication is crucial to parallel-computing applications such as distributed training of large-scale recommendation systems and natural language processing models. Existing collective communication libraries focus on optimizing operations for dense inputs, resulting in transmissions of many zeros when inputs are sparse. This counters current trends that see increasing data sparsity in large models. We propose OmniReduce, an efficient streaming aggregation system that exploits sparsity to maximize effective bandwidth use by sending only non-zero data blocks. We demonstrate that this idea is beneficial and accelerates distributed training by up to 8.2x. Even at 100 Gbps, OmniReduce delivers 1.4--2.9x better performance for network-bottlenecked DNNs.Keywords
Funding Information
- King Abdullah University of Science and Technology (OSR-CRG2020-4382)
This publication has 19 references indexed in Scilit:
- Parameter Hub2018
- Deep Contextualized Word Representations2018
- In-Network Computation is a Dumb Idea Whose Time Has Come2017
- Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- P4ACM SIGCOMM Computer Communication Review, 2014
- Bandwidth optimal all-reduce algorithms for clusters of workstationsJournal of Parallel and Distributed Computing, 2009
- MPI Reduction Operations for Sparse Floating-point DataLecture Notes in Computer Science, 2008
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005
- Stochastic Estimation of the Maximum of a Regression FunctionThe Annals of Mathematical Statistics, 1952