Configuring Distributed Computations Using Response Surfaces
Authors:
Open Access
- 24 November 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 235-246
Abstract
Configuring large distributed computations is a challenging task. Efficiently executing distributed computations requires configuration tuning based on careful examination of application and hardware properties. Considering the large number of parameters and impracticality of using trial and error in a production environment, programmers tend to make these decisions based on their experience and rules of thumb. Such configurations can lead to underutilized and costly clusters, and missed deadlines. In this paper, we present a new methodology for determining desired hardware and software configuration parameters for distributed computations. The key insight behind this methodology is to build a response surface that captures how applications perform under different hardware and software configuration. Such a model can be built through iterated experiments using the real system, or, more efficiently, using a simulator. The resulting model can then generate recommendations for configuration parameters that are likely to yield the desired results even if they have not been tried either in simulation or in real-life. The process can be iterated to refine previous predictions and achieve better results. We have implemented this methodology in a configuration recommendation system for MapReduce 2.0 applications. Performance measurements show that representative applications achieve up to 5× performance improvement when they use the recommended configuration parameters compared to the default ones.Keywords
Funding Information
- Air Force Office of Scientific Research (FA95550-11-1-0137)
- National Science Foundation (CCF-0424422, CNS-1111698, CNS-1518779, CCF-1116298)
- Defense Advanced Research Projects Agency (FA8750-11-2-0256)
- U.S. Department of Energy (DE-SC0006791)
This publication has 34 references indexed in Scilit:
- Apache Hadoop YARN2013
- Gunther: Search-Based Auto-Tuning of MapReduce2013
- HSim: A MapReduce simulator in enabling Cloud ComputingFuture Generation Computer Systems, 2013
- Experience transfer for the configuration tuning in large scale computing systemsACM SIGMETRICS Performance Evaluation Review, 2009
- Response surface methodology (RSM) as a tool for optimization in analytical chemistryTalanta, 2008
- DryadACM SIGOPS Operating Systems Review, 2007
- Application of response surface methodology in the optimization of cutting conditions for surface roughnessJournal of Materials Processing Technology, 2005
- An Approach to Performance Prediction for Parallel ApplicationsLecture Notes in Computer Science, 2005
- GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computingConcurrency and Computation: Practice and Experience, 2002
- Completely Derandomized Self-Adaptation in Evolution StrategiesEvolutionary Computation, 2001