High Throughput Intra-Node MPI Communication with Open-MX
Author:
- 1 January 2009
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10666192,p. 173-180
Abstract
The increasing number of cores per node in high-performance computing requires an efficient intra-node MPI communication subsystem. Most existing MPI implementations rely on two copies across a shared memory-mapped file. Open-MX offers a single-copy mechanism that is tightly integrated in its regular communication stack, making it transparently available to the MX backend of many MPI layers. We describe this implementation and its offloaded copy backend using I/OAT hardware. Memory pinning requirements are then discussed, and overlapped pinning is introduced to enable the start of Open-MX intra-node data transfer earlier. Performance evaluation shows that this local communication stack performs better than MPICH2 and Open-MPI for large messages, reaching up to 70% better throughput in micro-benchmarks when using I/OAT copy offload. Thanks to a single-copy being involved, the Open-MX intra-node communication throughput also does not heavily depend on cache sharing between processing cores, making these performance improvements easier to observe in real applications.Keywords
This publication has 9 references indexed in Scilit:
- Improving message passing over Ethernet with I/OAT copy offload in Open-MX2008
- Design and implementation of Open-MX: High-performance message passing over generic Ethernet hardware2008
- Designing Efficient Asynchronous Memory Operations Using Hardware Copy Engine: A Case Study with I/OAT2007
- Efficient asynchronous memory copy operations on multi-core systems and I/OAT2007
- Implementation and Shared-Memory Evaluation of MPICH2 over the Nemesis Communication Subsystem2006
- Designing High Performance and Scalable MPI Intra-node Communication Support for Clusters2006
- Optimizations of client's side communications in a distributed file system within a Myrinet cluster2004
- MPI versus MPI+OpenMP on the IBM SP for the NAS Benchmarks2000
- The Nas Parallel BenchmarksThe International Journal of Supercomputing Applications, 1991