Progress-Aware Transmission Protocol for Efficient In-Network Aggregation in Distributed Machine Learning
- 27 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Networking
- Vol. 33 (6) , 2897-2912
Abstract
Large-scale machine learning typically adopts distributed machine learning (DML) techniques to accelerate model training. Due to the large communication overhead, unfortunately, the phase of gradient aggregation has become the performance bottleneck for data-parallel DML. To reduce traffic volume, several in-network aggregation (INA) transmission protocols are proposed to offload gradient aggregation function into the programmable switches. However, since existing INA transmission protocols use synchronous congestion control mechanism to drive each round of gradient aggregation, the straggling workers lead to long iteration time and significant performance degradation. Besides, we reveal that existing INA solutions cannot provide the fairness performance among multiple jobs with varying number of workers. To solve the above problem, we propose PA-ATP, a progress-aware INA transmission protocol, which adopts the progress-aware asynchronous congestion control. PA-ATP adjusts the sending rate in accordance with the transmission progress, allowing the straggling flow to grab more bandwidth than the leading flow and control the asynchronous degree of straggling job. Moreover, to ensure the fair throughput among multiple jobs, we dynamically adjust the aggregator allocation for each job by tuning the number of hash operations. We use a P4 programmable switch and a kernel-bypass protocol stack to implement PA-ATP. The results of testbed and large-scale NS3 simulations show that PA-ATP reduces training time by up to 62% compared to the state-of-the-art INA transmission protocols.Keywords
Funding Information
- National Natural Science Foundation of China (U24A20245)
- National Natural Science Foundation of China (62132022)
- National Natural Science Foundation of China (62302524)
- National Natural Science Foundation of China (624722451)
- science and technology innovation Program of Hunan Province (2024RC1005)
- Natural Science Foundation of Hunan Province (2024JJ5050)
- Natural Science Foundation of Hunan Province (2024JJ6531)
This publication has 22 references indexed in Scilit:
- Straggler-Aware Gradient Aggregation for Large-Scale Distributed Deep Learning SystemIEEE/ACM Transactions on Networking, 2024
- Accelerating Distributed Training With Collaborative In-Network AggregationIEEE/ACM Transactions on Networking, 2024
- Preemptive Switch Memory Usage to Accelerate Training Jobs with Shared In-Network Aggregation2023
- PA-ATP: Progress-Aware Transmission Protocol for In-Network Aggregation2023
- Enabling Switch Memory Management for Distributed Training with In-Network Aggregation2023
- GRID: Gradient Routing With In-Network Aggregation for Distributed TrainingIEEE/ACM Transactions on Networking, 2023
- Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)TM Streaming-Aggregation Hardware Design and Evaluation2020
- Speeding Up Distributed Machine Learning Using CodesIEEE Transactions on Information Theory, 2017
- Scalable Hierarchical Aggregation Protocol (SHArP): A Hardware Architecture for Efficient Data Reduction2016
- Deep Residual Learning for Image Recognition2016