Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning
Top Cited Papers
Open Access
- 27 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 178-191
Abstract
No abstract availableThis publication has 19 references indexed in Scilit:
- DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining2023
- Chimera2021
- Efficient large-scale language model training on GPU clusters using megatron-LM2021
- Exploiting Simultaneous Communications to Accelerate Data Parallel Distributed Deep Learning2021
- DAPPLE2021
- Synthesizing optimal collective algorithms2021
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- DeepSpeed2020
- A generic communication scheduler for distributed DNN training acceleration2019
- Improving the Performance of Collective Operations in MPICH2003