Framework for Scalable Intra-Node Collective Operations using Shared Memory
Authors:
- Surabhi JainSJSurabhi Jain
- Rashid KaleemRKRashid Kaleem
- Marc Gamell BalmanaMBMarc Gamell Balmana
- Akhil LangerALAkhil Langer
- Dmitry DurnovDDDmitry Durnov
- Alexander SannikovASAlexander Sannikov
- Maria GarzaranMGMaria Garzaran
- 1 November 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 374-385
Abstract
Collective operations are used in MPI programs to express common communication patterns, collective computations, or synchronization. In many collectives, such as MPI_Allreduce, the intra-node component of the collective lies on the critical path, as the inter-node communication cannot start until the intra-node component has completed. With increasing number of core counts in each node, intra-node optimizations that leverage shared memory become more important. In this paper, we focus on the performance benefit of optimizing intra-node collectives using POSIX shared memory for synchronization and data sharing. We implement several collectives using basic primitives or steps as building blocks. Key components of our implementation include a dedicated intra- node collectives layer, careful layout of the data structures, as well as optimizations to exploit the memory hierarchy to balance parallelism and latencies of data movement. A comparison of our implementation on top of MPICH shows significant performance speedups with respect to the original MPICH implementation, MVAPICH, and OpenMPI.Keywords
This publication has 11 references indexed in Scilit:
- Why is MPI so slow?2017
- NUMA-aware shared-memory collective communication for MPI2013
- Kernel Assisted Collective Intra-node MPI Communication among Multi-Core and Many-Core CPUs2011
- Optimal bucket algorithms for large MPI collectives on torus interconnects2010
- Designing topology-aware collective communication algorithms for large scale InfiniBand clusters: Case studies with Scatter and Gather2010
- Cache-Efficient, Intranode, Large-Message MPI Communication with MPICH2-Nemesis2009
- MPI Support for Multi-core Architectures: Optimized Shared Memory Collectives2008
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005
- Fast collective operations using shared and remote memory access protocols on clusters2004
- Algorithms for scalable synchronization on shared-memory multiprocessorsACM Transactions on Computer Systems, 1991