Locality-Aware Reduce Task Scheduling for MapReduce
Authors:
- Mohammad HammoudMHMohammad Hammoud
- Majd F. SakrMSMajd F. Sakr
- 1 November 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 570-576
Abstract
MapReduce offers a promising programming model for big data processing. Inspired by functional languages, MapReduce allows programmers to write functional-style code which gets automatically divided into multiple map and/or reduce tasks and scheduled over distributed data across multiple machines. Hadoop, an open source implementation of MapReduce, schedules map tasks in the vicinity of their inputs in order to diminish network traffic and improve performance. However, Hadoop schedules reduce tasks at requesting nodes without considering data locality leading to performance degradation. This paper describes Locality-Aware Reduce Task Scheduler (LARTS), a practical strategy for improving MapReduce performance. LARTS attempts to collocate reduce tasks with the maximum required data computed after recognizing input data network locations and sizes. LARTS adopts a cooperative paradigm seeking a good data locality while circumventing scheduling delay, scheduling skew, poor system utilization, and low degree of parallelism. We implemented LARTS in Hadoop-0.20.2. Evaluation results show that LARTS outperforms the native Hadoop reduce task scheduler by an average of 7%, and up to 11.6%.Keywords
This publication has 11 references indexed in Scilit:
- LEEN: Locality/Fairness-Aware Key Partitioning for MapReduce in the Cloud2010
- LEMO-MR: Low Overhead and Elastic MapReduce Implementation Optimized for Memory and CPU-Intensive Applications2010
- Skew-resistant parallel processing of feature-extracting scientific user-defined functions2010
- The HiBench benchmark suite: Characterization of the MapReduce-based data analysis2010
- Cloud technologies for bioinformatics applications2009
- CLOUDLET2009
- Supporting MapReduce on large-scale asymmetric multi-core clustersACM SIGOPS Operating Systems Review, 2009
- HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment2009
- Mars2008
- The Google file system2003