An Analysis of Traces from a Production MapReduce Cluster
Authors:
- Soila KavulyaSKSoila Kavulya
- Jiaqi TanJTJiaqi Tan
- Rajeev GandhiRGRajeev Gandhi
- Priya NarasimhanPNPriya Narasimhan
- 1 January 2010
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 94-103
Abstract
MapReduce is a programming paradigm for parallel processing that is increasingly being used for data-intensive applications in cloud computing environments. An understanding of the characteristics of workloads running in MapReduce environments benefits both the service providers in the cloud and users: the service provider can use this knowledge to make better scheduling decisions, while the user can learn what aspects of their jobs impact performance. This paper analyzes 10-months of MapReduce logs from the M45 supercomputing cluster which Yahoo! made freely available to select universities for academic research. We characterize resource utilization patterns, job patterns, and sources of failures. We use an instance-based learning technique that exploits temporal locality to predict job completion times from historical data and identify potential performance problems in our dataset.Keywords
This publication has 15 references indexed in Scilit:
- MapReduceCommunications of the ACM, 2008
- A large-scale study of failures in high-performance computing systems2006
- Estimating computation times of data-intensive applicationsIEEE Distributed Systems Online, 2004
- Failure data analysis of a large-scale heterogeneous server environment2004
- The workload on parallel supercomputers: modeling the characteristics of rigid jobsJournal of Parallel and Distributed Computing, 2003
- Predictive application-performance modeling in a computational grid environment2003
- The elusive goal of workload characterizationACM SIGMETRICS Performance Evaluation Review, 1999
- A census of Tandem system availability between 1985 and 1990IEEE Transactions on Reliability, 1990
- Collinearity and Least Squares RegressionStatistical Science, 1987
- Measurement and modeling of computer reliability as affected by system activityACM Transactions on Computer Systems, 1986