Reliability analysis of deduplicated and erasure-coded storage
Authors:
Open Access
- 3 January 2011
- journal article
- Published by Association for Computing Machinery (ACM) in ACM SIGMETRICS Performance Evaluation Review
- Vol. 38 (3) , 4-9
Abstract
Space efficiency and data reliability are two primary concerns for modern storage systems. Chunk-based deduplication, which breaks up data objects into single-instance chunks that can be shared across objects, is an effective method for saving storage space. However, deduplication affects data reliability because an object's constituent chunks are often spread across a large number of disks, potentially decreasing the object's reliability. Therefore, an important problem in deduplicated storage is how to achieve space efficiency yet maintain each object's original reliability. In this paper, we present initial results on the reliability analysis of HP-KVS, a deduplicated key-value store that allows each object to specify its own reliability level and that uses software erasure coding for data reliability. The combination of deduplication and erasure coding gives rise to several interesting research problems. We show how to compare the reliability of erasure codes with different parameters and how to analyze the reliability of a big data object given its constituent parts' reliabilities. We also present a method for system designers to determine under what conditions deduplication will save space for erasure-coded data.Keywords
This publication has 7 references indexed in Scilit:
- Efficient eventual consistency in Pahoehoe, an erasure-coded key-blob archive2010
- R-ADMAD2009
- Understanding disk failure ratesACM Transactions on Storage, 2007
- An analysis of latent sector errors in disk drives2007
- Mirrored Disk Organization Reliability AnalysisIEEE Transactions on Computers, 2006
- Providing High Reliability in a Minimum Redundancy Archival Storage System2006
- A case for redundant arrays of inexpensive disks (RAID)1988