Designing high performance communication runtime for GPU managed memory
Open Access
- 12 March 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 82-91
Abstract
Graphics Processing Units (GPUs) have gained the position of a main stream accelerator due to its low power footprint and massive parallelism. CUDA 6.0 onward, NVIDIA has introduced the Managed Memory capability which unifies the host and device memory allocations into a single allocation and removes the requirement for explicit memory transfers between either memories. Several applications particularly of irregular nature can have immense benefits from managed memory because of the high productivity in programming that can be achieved owing to the minimal effort involved in the data management and movement. The MVAPICH2 library utilizes runtime designs such as CUDA Inter Process Communications (IPC) and GPUDirect RDMA (GDR) under the CUDA-Aware concept, to offer high productivity and programmability with MPI on modern clusters. However, integration and interaction of managed memory with these features raises challenges for efficient small and large message communications. In this study, we present an initial evaluation of managed memory capability and its interaction with existing high performance designs and features available in MVAPICH2 library. We propose new designs to enable efficient communication support between managed memory buffers. We also perform fine tuning to optimize the transfers between managed memories residing in GPUs. To the best of our knowledge, this is the first evaluation and study of managed memory and its interaction with MPI runtimes. A detailed evaluation and analysis of the performance of the proposed designs is presented. The Stencil2D communication kernel available in the SHOC suite was re-designed to enable the managed memory support. The evaluation shows a 4x improvement in the timings of stencil exchanges on 16 GPU nodes.Keywords
Funding Information
- National Science Foundation (OCI-1148371)
This publication has 10 references indexed in Scilit:
- Designing efficient small message transfer mechanism for inter-node MPI communication on InfiniBand GPU clusters2014
- An investigation of Unified Memory Access performance in CUDA2014
- HAND: A Hybrid Approach to Accelerate Non-contiguous Data Movement Using MPI Datatypes on GPU Clusters2014
- Efficient Inter-node MPI Communication Using GPUDirect RDMA for InfiniBand Clusters with NVIDIA GPUs2013
- Optimizing MPI Communication on Multi-GPU Systems Using CUDA Inter-Process Communication2012
- FLAT2012
- Stencil computations on heterogeneous platforms for the Jacobi method: GPUs versus Cell BEThe Journal of Supercomputing, 2012
- Physis2011
- Optimized Non-contiguous MPI Datatype Communication for GPU Clusters: Design, Implementation and Evaluation with MVAPICH22011
- MVAPICH2-GPU: optimized GPU to GPU communication for InfiniBand clustersComputer Science - Research and Development, 2011