MCCL: Measurement-based Collective Communication Library
Open Access
- 19 November 2026
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Networking
- Vol. 4 (CoNEXT4) , 1-25
Abstract
Collective communication is a critical performance bottleneck in large-scale distributed training. Existing libraries and synthesis frameworks compute open-loop schedules once, assuming a static topology and peak link capacities. In production, however, persistent gray failures can silently degrade link throughput and latency for hours, causing pre-computed schedules to lose much of their efficiency during execution. In this paper, we present Measurement-based Collective Communication Library (MCCL), a collective communication library that closes the loop between runtime network measurement and dynamic schedule synthesis. MCCL passively measures GPU-to-GPU throughput, detects persistent degradation, re-synthesizes the affected communication-intensive schedules with warm-started synthesizers, and hot-swaps them without interrupting training. We validate the complete loop on a multi-node hardware prototype and use simulations to evaluate cluster-scale scenarios. In our experiments, MCCL incurs negligible overhead (<2%) while maintaining high measurement accuracy (<1.2% error) and low schedule hot-swap time (≈ 12 ms). Notably, under gray failures, MCCL reduces training time by up to 4.99x over static schedules on simulated training workloads for state-of-the-art LLMs such as Llama and DeepSeek.This publication has 18 references indexed in Scilit:
- Crux: GPU-Efficient Communication Scheduling for Deep Learning Training2024
- Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem2024
- xCCL: A Survey of Industry-Led Collective Communication Libraries for Deep LearningJournal of Computer Science and Technology, 2023
- Routing and slot allocation in 5G hard slicingComputer Communications, 2023
- FAst in-network GraY failure detection for ISPs2022
- Jupiter evolving2022
- Synthesizing optimal collective algorithms2021
- Fail-Slow at ScaleACM Transactions on Storage, 2018
- Gray Failure2017
- The ellipsoid method and its consequences in combinatorial optimizationCombinatorica, 1981