ZHT: A Light-Weight Reliable Persistent Dynamic Scalable Zero-Hop Distributed Hash Table
Authors:
- Tonglin LiTLTonglin Li
- Xiaobing ZhouXZXiaobing Zhou
- Kevin BrandstatterKBKevin Brandstatter
- Dongfang ZhaoDZDongfang Zhao
- Ke WangKWKe Wang
- Anupam RajendranARAnupam Rajendran
- Zhao ZhangZZZhao Zhang
- Ioan RaicuIRIoan Raicu
- 1 May 2013
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15302075,p. 775-787
Abstract
This paper presents ZHT, a zero-hop distributed hash table, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. The goals of ZHT are delivering high availability, good fault tolerance, high throughput, and low latencies, at extreme scales of millions of nodes. ZHT has some important properties, such as being light-weight, dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append (providing lock-free concurrent key/value modifications) in addition to insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 512-cores, to an IBM Blue Gene/P supercomputer with 160K-cores. Using micro-benchmarks, we scaled ZHT up to 32K-cores with latencies of only 1.1ms and 18M operations/sec throughput. This work provides three real systems that have integrated with ZHT, and evaluate them at modest scales. 1) ZHT was used in the FusionFS distributed file system to deliver distributed meta-data management at over 60K operations (e.g. file create) per second at 2K-core scales. 2) ZHT was used in the IStore, an information dispersal algorithm enabled distributed object storage system, to manage chunk locations, delivering more than 500 chunks/sec at 32-nodes scales. 3) ZHT was also used as a building block to MATRIX, a distributed job scheduling system, delivering 5000 jobs/sec throughputs at 2K-core scales. We compared ZHT against other distributed hash tables and key/value stores and found it offers superior performance for the features and portability it supports.Keywords
This publication has 23 references indexed in Scilit:
- DataSpaces2010
- Middleware support for many-task computingCluster Computing, 2010
- The case for a versatile storage systemACM SIGOPS Operating Systems Review, 2010
- Toward loosely coupled programming on petascale systems2008
- Cycloid: A constant-degree and lookup-efficient P2P overlay networkPerformance Evaluation, 2006
- BMI: a network abstraction layer for parallel I/O2005
- Looking up data in P2P systemsCommunications of the ACM, 2003
- The Google file system2003
- Trans-dichotomous algorithms for minimum spanning trees and shortest paths2002
- A randomized linear-time algorithm to find minimum spanning treesJournal of the ACM, 1995