In-network Allreduce with Multiple Spanning Trees on PolarFly
Open Access
- 17 June 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 165-176
Abstract
Allreduce is a fundamental collective used in parallel computing and distributed training of machine learning models, and can become a performance bottleneck on large systems. In-network computing improves Allreduce performance by reducing packets on the fly using network routers. However, the throughput of current innetwork solutions is limited to a single link bandwidth. We develop, compare and contrast two different sets of Allreduce spanning trees embedded into PolarFly, a high-performance diameter-2 network topology. Both of our solutions offer theoretically guaranteed near-optimal performance, boosting Allreduce bandwidth by a factor equal to half the network radix of nodes. While our first set offers low-latency with trees of depth-3, the second set offers congestion-free implementation which reduces complexity and resource requirements of in-network computing units. In doing so, we also distinguish PolarFly as a highly suitable network for distributed deep learning and other applications that employ throughput-bound large Allreductions.Keywords
Funding Information
- Los Alamos National Laboratory (89233218CNA000001, 20230692ER)
- Intelligence Advanced Research Projects Activity (W911NF22C0081)
This publication has 38 references indexed in Scilit:
- Bandwidth optimal all-reduce algorithms for clusters of workstationsJournal of Parallel and Distributed Computing, 2009
- Optimizing a conjugate gradient solver with non-blocking collective operationsParallel Computing, 2007
- Performance analysis of MPI collective operationsCluster Computing, 2007
- Optimization of Collective Reduction Operations2004
- Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation2004
- Improving the Performance of Collective Operations in MPICH2003
- Finite fields: Encyclopedia of mathematics and its applicationsComputers & Mathematics with Applications, 1997
- On Graphs that do not Contain a Thomsen GraphCanadian Mathematical Bulletin, 1966
- Finite projective plane geometries and difference setsTransactions of the American Mathematical Society, 1953
- A theorem in finite projective geometry and some applications to number theoryTransactions of the American Mathematical Society, 1938