Enabling Efficient and Reliable Transition from Replication to Erasure Coding for Clustered File Systems
- 6 March 2017
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 28 (9) , 2500-2513
Abstract
To balance performance and storage efficiency, modern clustered file systems often first store data with replication, followed by encoding the replicated data with erasure coding. We argue that the commonly used random replication does not take into account erasure coding in its design, thereby raising both performance and availability issues in the subsequent encoding operation. We propose encoding-aware replication, which carefully places the replicas so as to (i) eliminate cross-rack downloads of data blocks during the encoding operation, (ii) preserve availability without data relocation after the encoding operation, and (iii) maintain load balancing across replicas as in random replication before the encoding operation. We conduct extensive HDFS-based testbed experiments and discrete-event simulations, and demonstrate the performance gains of encoding-aware replication over random replication.Keywords
Funding Information
- University Grants Committee
- Research Grants Council (AoE/E-02/08, CRF-C7036-15)
- National Natural Science Foundation of China (61502191, 61502190)
- ZTE Corporation
- Hubei Provincial Natural Science Foundation (2016CFB226)
This publication has 21 references indexed in Scilit:
- Enabling Efficient and Reliable Transition from Replication to Erasure Coding for Clustered File Systems2015
- Enabling Concurrent Failure Recovery for Regenerating-Coding-Based Storage Systems: From Theory to PracticeIEEE Transactions on Computers, 2014
- A "hitchhiker's" guide to fast and efficient data reconstruction in erasure-coded data centersACM SIGCOMM Computer Communication Review, 2014
- Lazy Means Smart2014
- Degraded-First Scheduling for MapReduce in Erasure-Coded Storage Clusters2014
- XORing elephantsProceedings of the VLDB Endowment, 2013
- Simple regenerating codes: Network coding for cloud storage2012
- Windows Azure Storage2011
- Erasure Coding Vs. Replication: A Quantitative Comparison2002
- Polynomial Codes Over Certain Finite FieldsJournal of the Society for Industrial and Applied Mathematics, 1960