Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem
Top Cited Papers
Open Access
- 4 August 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 16-37
Abstract
Cloud operators utilize collective communication optimizers to enhance the efficiency of the single-tenant, centrally managed training clusters they manage. However, current optimizers struggle to scale for such use cases and often compromise solution quality for scalability. Our solution, TE-CCL, adopts a traffic-engineering-based approach to collective communication. Compared to a state-of-the-art optimizer, TACCL, TE-CCL produced schedules with 2× better performance on topologies TACCL supports (and its solver took a similar amount of time as TACCL's heuristic-based approach). TECCL additionally scales to larger topologies than TACCL. On our GPU testbed, TE-CCL outperformed TACCL by 2.14× and RCCL by 3.18× in terms of algorithm bandwidth.Keywords
All Related Versions
Funding Information
- NSF (National Science Foundation) (CNS-2107147)
- Microsoft Research
- University of Pennsylvania
- University of Washington
This publication has 18 references indexed in Scilit:
- Synthesizing optimal collective algorithms2021
- Calendaring for wide area networks2014
- B4ACM SIGCOMM Computer Communication Review, 2013
- Achieving high utilization with software-driven WAN2013
- Inter-datacenter bulk transfers with netstitcher2011
- The Floyd–Warshall algorithm on graphs with negative cyclesInformation Processing Letters, 2010
- A survey of combinatorial optimization problems in multicast routingComputers & Operations Research, 2005
- How bad is naive multicast routing?2002
- Optimum routing of multicast streams2002
- A Formal Basis for the Heuristic Determination of Minimum Cost PathsIEEE Transactions on Systems Science and Cybernetics, 1968