Liangyu Zhao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Liangyu Zhao has a total of 12 co-authors. Here are the top co-authors by number of shared publications:
- Arvind Krishnamurthy
University of Washington, Seattle, USA
1 shared publication - Behnaz Arzani
Microsoft Research
1 shared publication - Jason Fantl
RTX BBN Technologies, Cambridge, MA, USA
1 shared publication - Joud Khoury
RTX BBN Technologies, Cambridge, MA, United States
1 shared publication - Luke Marshall
Microsoft Research, Redmond, USA
1 shared publication - Miguel Castro
OpenAI, San Francisco, USA
1 shared publication
Latest Publications
- Top Cited PapersOpen Access
- 4 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 16-37
Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem
Cloud operators utilize collective communication optimizers to enhance the efficiency of the single-tenant, centrally managed training clusters they manage. However, current optimizers struggle to scale for such use cases and often compromise solution quality for scalability. Our solution, TE-CCL, adopts a traffic-engineering-based approach to collective communication. Compared to a state-of-the-art optimizer, TACCL, TE-CCL produced schedules with 2× better performance on topologies TACCL supports (and its solver took a similar amount of time as TACCL's heuristic-based approach). TECCL additionally scales to larger topologies than TACCL. On our GPU testbed, TE-CCL outperformed TACCL by 2.14× and RCCL by 3.18× in terms of algorithm bandwidth.
- 4 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 16-37
Top Cited PapersOpen AccessRethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem
Cloud operators utilize collective communication optimizers to enhance the efficiency of the single-tenant, centrally managed training clusters they manage. However, current optimizers struggle to scale for such use cases and often compromise solution quality for scalability. Our solution, TE-CCL, adopts a traffic-engineering-based approach to collective communication. Compared to a state-of-the-art optimizer, TACCL, TE-CCL produced schedules with 2× better performance on topologies TACCL supports (and its solver took a similar amount of time as TACCL's heuristic-based approach). TECCL additionally scales to larger topologies than TACCL. On our GPU testbed, TE-CCL outperformed TACCL by 2.14× and RCCL by 3.18× in terms of algorithm bandwidth.
- Open Access
- 3 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 28-41
Efficient all-to-all Collective Communication Schedules for Direct-connect Topologies
The all-to-all collective communications primitive is widely used in machine learning (ML) and high performance computing (HPC) workloads, and optimizing its performance is of interest to both ML and HPC communities. All-to-all is a particularly challenging workload that can severely strain the underlying interconnect bandwidth at scale. This paper takes a holistic approach to optimize the performance of all-to-all collective communications on supercomputer-scale direct-connect interconnects. We address several algorithmic and practical challenges in developing efficient and bandwidth-optimal all-to-all schedules for any topology and lowering the schedules to various runtimes and interconnect technologies. We also propose a novel topology that delivers near-optimal all-to-all performance.
- 3 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 28-41
Open AccessEfficient all-to-all Collective Communication Schedules for Direct-connect Topologies
The all-to-all collective communications primitive is widely used in machine learning (ML) and high performance computing (HPC) workloads, and optimizing its performance is of interest to both ML and HPC communities. All-to-all is a particularly challenging workload that can severely strain the underlying interconnect bandwidth at scale. This paper takes a holistic approach to optimize the performance of all-to-all collective communications on supercomputer-scale direct-connect interconnects. We address several algorithmic and practical challenges in developing efficient and bandwidth-optimal all-to-all schedules for any topology and lowering the schedules to various runtimes and interconnect technologies. We also propose a novel topology that delivers near-optimal all-to-all performance.