Infiniband-Verbs on GPU: A Case Study of Controlling an Infiniband Network Device from the GPU
Authors:
- Lena OdenLOLena Oden
- Holger FroningHFHolger Froning
- Franz-Joseph PfreundtFPFranz-Joseph Pfreundt
- 1 May 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 976-983
Abstract
Due to their massive parallelism and high performance per watt GPUs gain high popularity in high performance computing and are a strong candidate for future exacscale systems. But communication and data transfer in GPU accelerated systems remain a challenging problem. Since the GPU normally is not able to control a network device, today a hybrid-programming model is preferred, whereby the GPU is used for calculation and the CPU handles the communication. As a result, communication between distributed GPUs suffers from unnecessary overhead, introduced by switching control flow from GPUs to CPUs and vice versa. In this work, we modify user space libraries and device drivers of GPUs and the Infiniband network device in a way to enable the GPU to control an Infiniband network device to independently source and sink communication requests without any involvements of the CPU. Our performance analysis shows the differences to hybrid communication models in detail, in particular that the CPU's advantage in generating work requests outshines the overhead associated with context switching. In other terms, our results show that complex networking protocols like IBVERBS are better handled by CPUs in spite of time penalties due to context switching, since overhead of work request generation cannot be parallelized and is not suitable with the high parallel programming model of GPUs.Keywords
This publication has 14 references indexed in Scilit:
- GGAS: Global GPU address spaces for efficient communication in heterogeneous clusters2013
- Direct MPI Library for Intel Xeon Phi Co-Processors2013
- Extending OpenSHMEM for GPU Computing2013
- BQCD with GPI: A case study2012
- Optimizing MPI Communication on Multi-GPU Systems Using CUDA Inter-Process Communication2012
- Extending MPI to accelerators2011
- GPU-to-CPU CallbacksLecture Notes in Computer Science, 2011
- Unified Parallel C for GPU Clusters: Language Extensions and Compiler Implementation2011
- The Fraunhofer virtual machine: a communication library and runtime system based on the RDMA modelComputer Science - Research and Development, 2009
- Message passing on data-parallel architectures2009