Efficient all-to-all Collective Communication Schedules for Direct-connect Topologies
Open Access
- 3 June 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 28-41
Abstract
The all-to-all collective communications primitive is widely used in machine learning (ML) and high performance computing (HPC) workloads, and optimizing its performance is of interest to both ML and HPC communities. All-to-all is a particularly challenging workload that can severely strain the underlying interconnect bandwidth at scale. This paper takes a holistic approach to optimize the performance of all-to-all collective communications on supercomputer-scale direct-connect interconnects. We address several algorithmic and practical challenges in developing efficient and bandwidth-optimal all-to-all schedules for any topology and lowering the schedules to various runtimes and interconnect technologies. We also propose a novel topology that delivers near-optimal all-to-all performance.Keywords
All Related Versions
Funding Information
- Defense Advanced Research Projects Agency (DARPA) (HR001120C0089)
This publication has 26 references indexed in Scilit:
- Distributed deep learning training using silicon photonic switched architecturesAPL Photonics, 2022
- X-NEST: A Scalable, Flexible, and High-Performance Network Architecture for Distributed Machine LearningJournal of Lightwave Technology, 2021
- Measuring and Understanding Throughput of Network Topologies2016
- Slim Fly: A Cost Effective Low-Diameter Network Topology2014
- Deadlock-Free Oblivious Routing for Arbitrary Topologies2011
- Efficient All-to-All Communication Patterns in Hypercube and Mesh Topologies2005
- All to-all communication with minimum start-up costs in 2D/3D tori and meshesIEEE Transactions on Parallel and Distributed Systems, 1998
- Optimum broadcasting and personalized communication in hypercubesIEEE Transactions on Computers, 1989
- Deadlock-Free Message Routing in Multiprocessor Interconnection NetworksIEEE Transactions on Computers, 1987
- A Design for Directed Graphs with Minimum DiameterIEEE Transactions on Computers, 1983