Adaptive Task Scheduling Strategy Based on Dynamic Workload Adjustment for Heterogeneous Hadoop Clusters
Authors:
- 12 June 2014
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Systems Journal
- Vol. 10 (2) , 471-482
Abstract
The original task scheduling algorithm of Hadoop cannot meet the performance requirements of heterogeneous clusters. According to the dynamic change of load of each task node and the difference of node performance of different tasks in the heterogeneous Hadoop cluster, a novel adaptive task scheduling strategy based on dynamic workload adjustment (ATSDWA) is presented. With ATSDWA, tasktrackers can adapt to the change of load at runtime, obtain tasks in accordance with the computing ability of their own, and realize the self-regulation, while avoiding the complexity of algorithm, which is the prime reason to make jobtracker the system performance bottleneck. Experimental results show that ATSDWA is a highly efficient and reliable algorithm, which can make heterogeneous Hadoop clusters stable, scalable, efficient, and load balancing. Furthermore, its performance is superior to the original and improved task scheduling strategy of Hadoop, from the aspects of the execution time of tasks, the resource utilization, and the speed-up ratio.Keywords
Funding Information
- National Natural Science Foundation of China (61202004, 61271233)
- China Postdoctoral Science Foundation (2011M500095, 2012T50514)
- Special Fund for Fast Sharing of Science Paper in Net Era by CSTD (2013116)
- Natural Science Foundation of Jiangsu Province (BK2011754)
- Natural Science Fund of Higher Education of Jiangsu Province (12KJB520007)
- State Key Laboratory for Novel Software Technology (2013116KFKT2012B21)
This publication has 14 references indexed in Scilit:
- A genetic algorithm for task scheduling on heterogeneous computing systems using multiple priority queuesInformation Sciences, 2014
- Energy-aware parallel task scheduling in a clusterFuture Generation Computer Systems, 2013
- A MapReduce task scheduling algorithm for deadline constraintsCluster Computing, 2012
- Adaptive energy-efficient scheduling for real-time tasks on DVS-enabled heterogeneous clustersJournal of Parallel and Distributed Computing, 2012
- Resource allocation and scheduling in cloud computing2012
- Windows Azure Platform Overview2011
- CloudBurst: highly sensitive read mapping with MapReduceBioinformatics, 2009
- Speculative execution in a distributed file systemACM SIGOPS Operating Systems Review, 2005
- A dynamic and reliability-driven scheduling algorithm for parallel real-time jobs executing on heterogeneous clustersJournal of Parallel and Distributed Computing, 2005
- Web search for a planet: the google cluster architectureIEEE Micro, 2003