A dynamic replica strategy based on Markov model for hadoop distributed file system (HDFS)
Authors:
- 19 December 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 337-342
Abstract
In this paper, we describe DRS, a dynamic replica strategy based on improved Markov model for Hadoop Distributed File System (HDFS). DRS can dynamically increase or reduce the replicas with the data becoming hot or cold. It includes two parts: a dynamic replica adjustment strategy (DRAS) and a homogeneous replica placement strategy (HRPS). First, DRS constructs a transition probability matrix based on the file accessing times in a period and calculates the stationary probability distribution of the system. It utilizes the results to distinguish different data types, and then increases extra replicas for hot data, clean up these extra replicas when the data cool down. Finally, these replicas are homogeneously distributed across the racks and nodes. The experimental results show that DRS is effective, and it is obviously superior to static replica strategy of HDFS.Keywords
This publication has 4 references indexed in Scilit:
- Adaptive Replication Management in HDFS Based on Supervised LearningIEEE Transactions on Knowledge and Data Engineering, 2016
- A Weight-Based Dynamic Replica Replacement Strategy in Data Grids2008
- Evaluating scheduling and replica optimisation strategies in OptorSim2004
- Web caching and Zipf-like distributions: evidence and implications1999