Accelerating Allreduce With In-Network Reduction on Intel PIUMA
- 29 December 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Micro
- Vol. 42 (2) , 44-52
Abstract
The Intel Programmable Integrated Unified Memory Architecture (PIUMA) system maps collective operations directly into the network switches and supports pipelined embeddings for high-throughput collective computation. Utilizing these features and PIUMA’s network topology, we develop a methodology to generate extremely low latency embeddings for in-network Allreduce. Our analysis shows that the proposed in-network Allreduce is highly scalable, with less than 1.5-μs single-element latency on 16K nodes. Compared to host-based Allreduce, it exhibits 36× less latency and 3.5× higher throughput. With deep neural network training as an example, we further demonstrate the benefits of our in-network Allreduce on end-user applications.Keywords
Funding Information
- Defense Advanced Research Projects Agency (HROOll17-3-0004)
This publication has 10 references indexed in Scilit:
- In-network reductions on multi-dimensional HyperX2021
- Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)TM Streaming-Aggregation Hardware Design and Evaluation2020
- An In-Network Architecture for Accelerating Shared-Memory Multiprocessor Collectives2020
- ImageNet Training in Minutes2018
- Parallel iterative refinement linear least squares solvers based on all-reduce operationsParallel Computing, 2016
- HyperX2009
- MPI Collective Communications on The Blue Gene/P Supercomputer: Algorithms and Optimizations2009
- Bandwidth optimal all-reduce algorithms for clusters of workstationsJournal of Parallel and Distributed Computing, 2009
- Optimization of Collective Reduction Operations2004
- Scalable NIC-based Reduction on Large-scale Clusters2003