Performance-driven task co-scheduling for MapReduce environments
- 1 January 2010
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15421201,p. 373-380
Abstract
MapReduce is a data-driven programming model proposed by Google in 2004 which is especially well suited for distributed data analytics applications. We consider the management of MapReduce applications in an environment where multiple applications share the same physical resources. Such sharing is in line with recent trends in data center management which aim to consolidate workloads in order to achieve cost and energy savings. In a shared environment, it is necessary to predict and manage the performance of workloads given a set of performance goals defined for them. In this paper, we address this problem by introducing a new task scheduler for a MapReduce framework that allows performance-driven management of MapReduce tasks. The proposed task scheduler dynamically predicts the performance of concurrent MapReduce jobs and adjusts the resource allocation for the jobs. It allows applications to meet their performance objectives without over-provisioning of physical resources.Keywords
This publication has 4 references indexed in Scilit:
- MapReduce optimization using regulated dynamic prioritization2009
- Scheduling despite inexact job-size information2008
- MapReduceCommunications of the ACM, 2008
- Adaptive and scalable comparison scheduling2007