Swapping to Remote Memory over InfiniBand: An Approach using a High Performance Network Block Device
- 1 September 2005
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in 2005 IEEE International Conference on Cluster Computing
- No. 15525244,p. 1-10
Abstract
Traditionally, operations with memory on other nodes (remote memory) in cluster environments interconnected with technologies like Gigabit Ethernet have been expensive with latencies several magnitudes slower than local memory accesses. Modern RDMA capable networks such as InfiniBand and Quadrics provide low latency of a few microseconds and high bandwidth of up to 10 Gbps. This has significantly reduced the latency gap between access to local memory and remote memory in modern clusters. Remote idle memory can be exploited to reduce the memory pressure on individual nodes. This is akin to adding an additional level in the memory hierarchy between local memory and the disk, with potentially dramatic performance improvements especially for memory intensive applications. In this paper, we take on the challenge to design a remote paging system for remote memory utilization in InfiniBand clusters. We present the design and implementation of a high performance networking block device (HPBD) over InfiniBand fabric, which serves as a swap device of kernel virtual memory (VM) system for efficient page transfer to/from remote memory servers. Our experiments show that using HPBD, quick sort performs only 1.45 times slower than local memory system, and up to 21 times faster than local disk. And our design is completely transparent to user applications. To the best of our knowledge, it is the first work of a remote pager design using InfiniBand for remote memory utilizationKeywords
This publication has 12 references indexed in Scilit:
- High Performance RDMA-Based MPI Implementation over InfiniBandInternational Journal of Parallel Programming, 2004
- Parallel network RAM: effectively utilizing global cluster memory for large data-intensive parallel programs2004
- Performance Evaluation of the Quadrics Interconnection NetworkCluster Computing, 2003
- Dodo: a user-level system for exploiting idle memory in workstation clusters2003
- Memory ushering in a scalable computing cluster2002
- The Network RamDisk: Using remote memory on heterogeneous NOWsCluster Computing, 1999
- Availability and utility of idle memory in workstation clusters1999
- Myrinet: a gigabit-per-second local area networkIEEE Micro, 1995
- Implementing global memory management in a workstation cluster1995
- The MOSIX Distributed Operating System1993