Using Pipeline Performance Prediction to Accelerate AutoML Systems
Open Access
- 18 June 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Abstract
Automatic machine learning (AutoML) systems aim to automate the synthesis of machine learning (ML) pipelines. An important challenge these systems face is how to efficiently search a large space of candidate pipelines. Several strategies have been proposed to navigate and prune the search space, from the use of grammars to deep learning models. However, regardless of the strategy used, a major overhead lies in the evaluation step: for each synthesized pipeline p, these systems must both train and test p to guide the search and to identify the best pipelines. Given a time budget and computing resources, the evaluation cost limits how much of the search space can be explored. As a result, these systems may miss good pipelines. We propose ML4ML, an approach that aims to reduce the evaluation overhead for AutoML systems. ML4ML leverages the provenance of prior pipeline runs to predict performance without having to re-train and test the pipelines. We present results of an experimental evaluation which demonstrates that not only can ML4ML build a reliable predictive model with low mean absolute error, but the integration of this model with AutoML systems leads to substantial speedups, enabling the systems to explore a larger number of pipelines and primitive combinations and derive pipelines at a much lower cost.Keywords
Funding Information
- NSF (IIS-2106888)
- DARPA (D3M program)
This publication has 20 references indexed in Scilit:
- OBOE2019
- TAPAS: Train-Less Accuracy Predictor for Architecture SearchProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Democratizing Data Science through Interactive Curation of ML Pipelines2019
- A Survey on Collecting, Managing, and Analyzing Provenance from ScriptsACM Computing Surveys, 2019
- Evolving Deep Neural Networks2019
- ML-Plan: Automated machine learning via hierarchical planningMachine Learning, 2018
- Learning Transferable Architectures for Scalable Image Recognition2018
- Genetic CNN2017
- Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn2014
- Auto-WEKA2013