Cache-Efficient, Intranode, Large-Message MPI Communication with MPICH2-Nemesis
Authors:
- Darius BuntinasDBDarius Buntinas
- Brice GoglinBGBrice Goglin
- David GoodellDGDavid Goodell
- Guillaume MercierGMGuillaume Mercier
- Stéphanie MoreaudSMStéphanie Moreaud
- 1 September 2009
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 462-469
Abstract
The emergence of multicore processors raises the need to efficiently transfer large amounts of data between local processes. MPICH2 is a highly portable MPI implementation whose large-message communication schemes suffer from high CPU utilization and cache pollution because of the use of a double-buffering strategy, common to many MPI implementations. We introduce two strategies offering a kernel-assisted, single-copy model with support for noncontiguous and asynchronous transfers. The first one uses the now widely available vmsplice Linux system call; the second one further improves performance thanks to a custom kernel module called KNEM. The latter also offers I/OAT copy offload, which is dynamically enabled depending on both hardware cache characteristics and message size. These new solutions outperform the standard transfer method in the MPICH2 implementation when no cache is shared between the processing cores or when very large messages are being transferred. Collective communication operations show a dramatic improvement, and the IS NAS parallel benchmark shows a 25% speedup and better cache efficiency.Keywords
This publication has 9 references indexed in Scilit:
- High Throughput Intra-Node MPI Communication with Open-MX2009
- Designing an Efficient Kernel-Level and User-Level Hybrid Approach for MPI Intra-Node Communication on Multi-Core Systems2008
- Performance Analysis and Evaluation of PCIe 2.0 and Quad-Data Rate InfiniBand2008
- Designing Efficient Asynchronous Memory Operations Using Hardware Copy Engine: A Case Study with I/OAT2007
- Lightweight kernel-level primitives for high-performance MPI intra-node communication over multi-core systems2007
- Efficient asynchronous memory copy operations on multi-core systems and I/OAT2007
- Data Transfers between Processes in an SMP System: Performance Study and Application to MPI2006
- Design and evaluation of Nemesis, a scalable, low-latency, message-passing communication subsystem2006
- A Portable Programming Interface for Performance Evaluation on Modern ProcessorsThe International Journal of High Performance Computing Applications, 2000