High-Performance Design of Hadoop RPC with RDMA over InfiniBand
Authors:
- Xiaoyi LuXLXiaoyi Lu
- Nusrat S. IslamNINusrat S. Islam
- Wasi- Ur- RahmanWRWasi- Ur- Rahman
- Jithin JoseJJJithin Jose
- Hari SubramoniHSHari Subramoni
- Hao WangHWHao Wang
- Dhabaleswar K. PandaDPDhabaleswar K. Panda
- 1 October 2013
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 01903918,p. 641-650
Abstract
Hadoop RPC is the basic communication mechanism in the Hadoop ecosystem. It is used with other Hadoop components like MapReduce, HDFS, and HBase in real world data-centers, e.g. Facebook and Yahoo!. However, the current Hadoop RPC design is built on Java sockets interface, which limits its potential performance. The High Performance Computing community has exploited high throughput and low latency networks such as InfiniBand for many years. In this paper, we first analyze the performance of current Hadoop RPC design by unearthing buffer management and communication bottlenecks, that are not apparent on the slower speed networks. Then we propose a novel design (RPCoIB) of Hadoop RPC with RDMA over InfiniBand networks. RPCoIB provides a JVM-bypassed buffer management scheme and utilizes message size locality to avoid multiple memory allocations and copies in data serialization and deserialization. Our performance evaluations reveal that the basic ping-pong latencies for varied data sizes are reduced by 42%-49% and 46%-50% compared with 10GigE and IPoIB QDR (32Gbps), respectively, while the RPCoIB design also improves the peak throughput by 82% and 64% compared with 10GigE and IPoIB. As compared to default Hadoop over IPoIB QDR, our RPCoIB design improves the performance of the Sort benchmark on 64 compute nodes by 15%, while it improves the performance of CloudBurst application by 10%. We also present thorough, integrated evaluations of our RPCoIB design with other research directions, which optimize HDFS and HBase using RDMA over InfiniBand. Compared with their best performance, we observe 10% improvement for HDFS-IB, and 24% improvement for HBase-IB. To the best of our knowledge, this is the first such design of the Hadoop RPC system over high performance networks such as InfiniBand.Keywords
This publication has 8 references indexed in Scilit:
- Understanding the communication characteristics in HBase: What are the fundamental bottlenecks?2012
- Memcached Design on High Performance RDMA Capable Interconnects2011
- Can MPI Benefit Hadoop and MapReduce Applications?2011
- Unifying UPC and MPI runtimes2010
- Benchmarking cloud serving systems with YCSB2010
- The Hadoop Distributed File System2010
- Differential serialization for optimized SOAP performance2004
- Efficient Java RMI for parallel programmingACM Transactions on Programming Languages and Systems, 2001