DARE: Adaptive Data Replication for Efficient Cluster Scheduling
Authors:
- Cristina L. AbadCACristina L. Abad
- Yi LuYLYi Lu
- Roy H. CampbellRCRoy H. Campbell
- 1 September 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15525244,p. 159-168
Abstract
Placing data as close as possible to computation is a common practice of data intensive systems, commonly referred to as the data locality problem. By analyzing existing production systems, we confirm the benefit of data locality and find that data have different popularity and varying correlation of accesses. We propose DARE, a distributed adaptive data replication algorithm that aids the scheduler to achieve better data locality. DARE solves two problems, how many replicas to allocate for each file and where to place them, using probabilistic sampling and a competitive aging algorithm independently at each node. It takes advantage of existing remote data accesses in the system and incurs no extra network usage. Using two mixed workload traces from Face book, we show that DARE improves data locality by more than 7 times with the FIFO scheduler in Hadoop and achieves more than 85% data locality for the FAIR scheduler with delay scheduling. Turnaround time and job slowdown are reduced by 19% and 25\%, respectively.Keywords
This publication has 19 references indexed in Scilit:
- CassandraACM SIGOPS Operating Systems Review, 2010
- Delay scheduling2010
- The Impact of Virtualization on Network Performance of Amazon EC2 Data Center2010
- The nature of data center traffic2009
- BigtableACM Transactions on Computer Systems, 2008
- Dryad2007
- The impact of data replication on job scheduling performance in the Data GridFuture Generation Computer Systems, 2006
- When the herd is smart: aggregate behavior in the selection of job requestIEEE Transactions on Parallel and Distributed Systems, 2003
- Metrics and benchmarking for parallel job scheduling1998
- A Survey of Distributed File SystemsAnnual Review of Computer Science, 1990