Design Considerations and Analysis of Multi-Level Erasure Coding in Large-Scale Data Centers
Open Access
- 11 November 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Abstract
Multi-level erasure coding (MLEC) has seen large deployments in the field, but there is no in-depth study of design considerations for MLEC at scale. In this paper, we provide comprehensive design considerations and analysis of MLEC at scale. We introduce the design space of MLEC in multiple dimensions, including various code parameter selections, chunk placement schemes, and various repair methods. We quantify their performance and durability, and show which MLEC schemes and repair methods can provide the best tolerance against independent/correlated failures and reduce repair network traffic by orders of magnitude. To achieve this, we use various evaluation strategies including simulation, splitting, dynamic programming, and mathematical modeling. We also compare the performance and durability of MLEC with other EC schemes such as SLEC and LRC and show that MLEC can provide high durability with higher encoding throughput and less repair network traffic over both SLEC and LRC.Keywords
This publication has 18 references indexed in Scilit:
- OI-RAID: A Two-Layer RAID Architecture towards Fast Recovery and High Reliability2016
- A Family of Optimal Locally Recoverable CodesIEEE Transactions on Information Theory, 2014
- Highly reliable two-dimensional RAID arrays for archival storage2012
- Enhancing SSD reliability through efficient RAID support2012
- Optimal real number codes for fault tolerant matrix operations2009
- Multi-level RAID for very large disk arraysACM SIGMETRICS Performance Evaluation Review, 2006
- Techniques for fast simulation of models of highly dependable systemsIEEE Transactions on Reliability, 2001
- Reliability and performance of hierarchical RAID with multiple controllers2001
- Tolerating multiple failures in RAID architectures with optimal storage and uniform declustering1997
- Splitting for rare event simulation1996