Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T)
Authors:
- Atrisha SarkarASAtrisha Sarkar
- Jianmei GuoJGJianmei Guo
- Norbert SiegmundNSNorbert Siegmund
- Sven ApelSASven Apel
- Krzysztof CzarneckiKCKrzysztof Czarnecki
- 1 November 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 342-352
Abstract
A key challenge of the development and maintenanceof configurable systems is to predict the performance ofindividual system variants based on the features selected. It isusually infeasible to measure the performance of all possible variants, due to feature combinatorics. Previous approaches predictperformance based on small samples of measured variants, butit is still open how to dynamically determine an ideal samplethat balances prediction accuracy and measurement effort. Inthis paper, we adapt two widely-used sampling strategies forperformance prediction to the domain of configurable systemsand evaluate them in terms of sampling cost, which considersprediction accuracy and measurement effort simultaneously. Togenerate an initial sample, we introduce a new heuristic based onfeature frequencies and compare it to a traditional method basedon t-way feature coverage. We conduct experiments on six realworldsystems and provide guidelines for stakeholders to predictperformance by sampling.Keywords
This publication has 15 references indexed in Scilit:
- Performance Prediction of Configurable Software Systems by Fourier Learning (T)2015
- Performance-influence models for highly configurable systems2015
- Automated inference of goal-oriented performance prediction functions2012
- Maximizing classifier utility when there are data acquisition and modeling costsData Mining and Knowledge Discovery, 2007
- Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems2005
- Model-based performance prediction in software development: a surveyIEEE Transactions on Software Engineering, 2004
- Improving Progressive Sampling via Meta-learning on Learning CurvesLecture Notes in Computer Science, 2004
- Data Reduction Using Multiple Models IntegrationLecture Notes in Computer Science, 2001
- Efficient progressive sampling1999
- Smoothing Techniques1991