Towards Automatic Tuning of Apache Spark Configuration
- 1 July 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 417-425
Abstract
Apache Spark provides a large number of configuration settings that may be tuned to improve the performance of specific applications running on the platform. However, it is non-trivial to identify the combination of settings that may improve the performance of a specific application as the influence of each setting on performance may vary across applications. As identifying the optimal combination of settings is computationally infeasible due to exponential search space, in this paper we investigate machine learning based approaches to construct application specific performance influence models, and use them to tune the performance of specific applications running on Apache Spark platform. We evaluated our approach using 9 different applications on a 6 node cluster and demonstrated that our framework can reduce execution time by 22.8% to 40.0% depending on applications.Keywords
This publication has 11 references indexed in Scilit:
- Understanding the Influence of Configuration Settings: An Execution Model-Driven Framework for Apache Spark Platform2017
- CSMiner: An Automated Tool for Analyzing Changes in Configuration Settings across Multiple Versions of Large Scale Cloud Software2016
- Performance-influence models for highly configurable systems2015
- MRONLINE2014
- Classification and regression treesWIREs Data Mining and Knowledge Discovery, 2011
- Boosting the performance of computing systems through adaptive configuration tuning2009
- Automatic configuration of internet services2007
- Finding probably better system configurations quicklyACM SIGMETRICS Performance Evaluation Review, 2006
- A tutorial on support vector regressionStatistics and Computing, 2004
- A recursive random search algorithm for large-scale network parameter configuration2003