Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE
- 30 March 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 47 (12) , 2956-2972
Abstract
As software systems grow in complexity and the space of possible configurations increases exponentially, finding the near-optimal configuration of a software system becomes challenging. Recent approaches address this challenge by learning performance models based on a sample set of configurations. However, collecting enough sample configurations can be very expensive since each such sample requires configuring, compiling, and executing the entire system using a complex test suite. When learning on new data is too expensive, it is possible to use Transfer Learning to “transfer” old lessons to the new context. Traditional transfer learning has a number of challenges, specifically, (a) learning from excessive data takes excessive time, and (b) the performance of the models built via transfer can deteriorate as a result of learning from a poor source. To resolve these problems, we propose a novel transfer learning framework called BEETLE, which is a “bellwether”-based transfer learner that focuses on identifying and learning from the most relevant source from amongst the old data. This paper evaluates BEETLE with 57 different software configuration problems based on five software systems (a video encoder, an SAT solver, a SQL database, a high-performance C-compiler, and a streaming data analytics tool). In each of these cases, BEETLE found configurations that are as good as or better than those found by other state-of-the-art transfer learners while requiring only a fraction ( $\frac{1}{7}$ th) of the measurements needed by those other methods. Based on these results, we say that BEETLE is a new high-water mark in optimally configuring software.
Keywords
All Related Versions
This publication has 56 references indexed in Scilit:
- Algorithm runtime prediction: Methods & evaluationArtificial Intelligence, 2014
- Faster Hoeffding Racing: Bernstein Races via Jackknife EstimatesLecture Notes in Computer Science, 2013
- Evaluating prediction systems in software project estimationInformation and Software Technology, 2012
- Practical performance models for complex, popular applicationsACM SIGMETRICS Performance Evaluation Review, 2010
- On the relative value of cross-company and within-company data for defect predictionEmpirical Software Engineering, 2009
- A systematic review of effect size in software engineering experimentsInformation and Software Technology, 2007
- Gaussian Processes in Machine Learning2004
- A simulation study of the model evaluation criterion mmreIEEE Transactions on Software Engineering, 2003
- Exploiting Task Relatedness for Multiple Task Learning2003
- A fast and elitist multiobjective genetic algorithm: NSGA-IIIEEE Transactions on Evolutionary Computation, 2002