High performance RDMA-based design of HDFS over InfiniBand
Authors:
- N. S. IslamNIN. S. Islam
- M. W. RahmanMRM. W. Rahman
- J. JoseJJJ. Jose
- R. RajachandrasekarRRR. Rajachandrasekar
- H. WangHWH. Wang
- H. SubramoniHSH. Subramoni
- C. MurthyCMC. Murthy
- D. K. PandaDPD. K. Panda
- 1 November 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-12
Abstract
Hadoop Distributed File System (HDFS) acts as the primary storage of Hadoop and has been adopted by reputed organizations (Facebook, Yahoo! etc.) due to its portability and fault-tolerance. The existing implementation of HDFS uses Javasocket interface for communication which delivers suboptimal performance in terms of latency and throughput. For dataintensive applications, network performance becomes key component as the amount of data being stored and replicated to HDFS increases. In this paper, we present a novel design of HDFS using Remote Direct Memory Access (RDMA) over InfiniBand via JNI interfaces. Experimental results show that, for 5GB HDFS file writes, the new design reduces the communication time by 87% and 30% over 1Gigabit Ethernet (1GigE) and IP-over-InfiniBand (IPoIB), respectively, on QDR platform (32Gbps). For HBase, the Put operation performance is improved by 26% with our design. To the best of our knowledge, this is the first design of HDFS over InfiniBand networks.Keywords
This publication has 17 references indexed in Scilit:
- Hadoop acceleration through network levitated merge2011
- Using the Gfarm File System as a POSIX Compatible Storage Platform for Hadoop MapReduce Applications2011
- Hadoop high availability through metadata replication2009
- Understanding intrinsic characteristics and system implications of flash memory based solid state drives2009
- RDMA over Ethernet — A preliminary study2009
- BigtableACM Transactions on Computer Systems, 2008
- Experimental Analysis of InfiniBand Transport Services on WAN2008
- PVFS over InfiniBand: design and performance evaluation2003
- Principled design of the modern Web architectureACM Transactions on Internet Technology, 2002
- Space/time trade-offs in hash coding with allowable errorsCommunications of the ACM, 1970