Dynamic Replication Policy on HDFS Based on Machine Learning Clustering
Open Access
- 22 February 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 18551-18559
Abstract
Data growth in recent years has been swift, leading to the emergence of big data science. Distributed File Systems (DFS) are commonly used to handle big data, like Google File System (GFS), Hadoop Distributed File System (HDFS), and others. The DFS should provide the availability of data and reliability of the system in case of failure. The DFS replicates the files in different locations to provide availability and reliability. These replications consume storage space and other resources. The importance of these files differs depending on how frequently they are used in the system. So some of these files do not deserve to replicate many times because it is unimportant in the system. This paper introduces a Dynamic Replication Policy using Machine Learning Clustering (DRPMLC) on HDFS, which uses Machine Learning to cluster the files into different groups and apply other replication policies to each group to reduce the storage consumption, improve the read and write operations time and keep the availability and reliability of HDFS as a High-Performance Distributed Computing (HPDC).Keywords
This publication has 23 references indexed in Scilit:
- Hierarchical Density-Based Clustering Using MapReduceIEEE Transactions on Big Data, 2019
- An improved technique for increasing availability in Big Data replicationFuture Generation Computer Systems, 2019
- Erasure coding for distributed storage: an overviewScience China Information Sciences, 2018
- Big data multi-query optimisation with Apache FlinkInternational Journal of Web Engineering and Technology, 2018
- Optimizing Join in HIVE Star Schema Using Key/Facts IndexingIETE Technical Review, 2017
- Benchmarking the performance of hadoop triple replication and erasure coding on a nation-wide distributed cloud2015
- JOMR: Multi-join optimizer technique to enhance map-reduce job2014
- ERMS: An Elastic Replication Management System for HDFS2012
- Research on k-means Clustering Algorithm: An Improved k-means Clustering Algorithm2010
- MapReduceCommunications of the ACM, 2008