Layout-aware Hardware-assisted Designs for Derived Data Types in MPI
- 1 December 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 302-311
Abstract
Modern MPI-based scientific applications frequently use derived datatypes (DDT) for inter-process communication. Designing scalable solutions capable of dynamically adapting themselves to the complex communication requirements posed by DDT-based applications bring forth several new challenges. In this work, we address these challenges and propose solutions to efficiently improve the performance of hardware-assisted datatype transfers. Further, we design a layout-aware DDT scheme that dynamically adapts the datatype processing to the communication requirements of the datatype layouts used by the application. The proposed layout-aware adaptive scheme is able to dynamically switch between different host-based and the proposed hardware-assisted schemes to deliver the best performance and scalability while hiding the communication overheads. The experimental evaluations on multiple HPC systems including Frontera at TACC and Expanse at SDSC show that our proposed designs achieve up to 22 % improvement in performance over state-of-the-art MPI libraries at the micro-benchmark level. We also evaluate our designs with various scientific application kernels such as MILC, WRF, and applications such as miniGhost and demonstrate up to 9 % improvement in performance at 128 nodes for the miniGhost application.Keywords
This publication has 10 references indexed in Scilit:
- Network-accelerated non-contiguous memory transfers2019
- FALCON: Efficient Designs for Zero-Copy MPI Datatype Processing on Emerging Architectures2019
- DAME: Runtime-compilation for data movementThe International Journal of High Performance Computing Applications, 2017
- High Performance MPI Datatype Support with User-Mode Memory Registration: Challenges, Designs, and Benefits2015
- Designing MPI Library with Dynamic Connected Transport (DCT) of InfiniBand: Early Experiences2014
- MPI datatype processing using runtime compilation2013
- Micro-applications for Communication Data Access Patterns and MPI Datatypes2012
- Improving MPI communication via data type fission2010
- Zero-Copy MPI Derived Datatype Communication over InfiniBandLecture Notes in Computer Science, 2004
- Flattening on the Fly: efficient handling of MPI derived datatypesLecture Notes in Computer Science, 1999