Multi-Objective Hyperparameter Optimization in Machine Learning—An Overview
Authors:
- Florian KarlFKFlorian Karl
- Tobias PielokTPTobias Pielok
- Julia MoosbauerJMJulia Moosbauer
- Florian PfistererFPFlorian Pfisterer
- Stefan CoorsSCStefan Coors
- Martin BinderMBMartin Binder
- Lennart SchneiderLSLennart Schneider
- Janek ThomasJTJanek Thomas
- Jakob RichterJRJakob Richter
- Michel LangMLMichel Lang
- Eduardo C. Garrido-MerchánEGEduardo C. Garrido-Merchán
- Juergen BrankeJBJuergen Branke
Open Access
- 12 December 2023
- journal article
- review article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Evolutionary Learning and Optimization
- Vol. 3 (4) , 1-50
Abstract
Hyperparameter optimization constitutes a large part of typical modern machine learning (ML) workflows. This arises from the fact that ML methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies from the domains of evolutionary algorithms and Bayesian optimization. We illustrate the utility of multi-objective optimization in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability, and robustness.Keywords
Funding Information
- Bavarian Ministry of Economic Affairs, Regional Development and Energy through the Center for Analytics–Data–Applications (ADA-Center) within the framework of BAYERN DIGITAL II (20-3410-2-9-8)
- German Federal Ministry of Education and Research (01IS18036A)
This publication has 173 references indexed in Scilit:
- Stochastic multi-objective optimization: a survey on non-scalarizing methodsAnnals of Operations Research, 2013
- Robustness and generalizationMachine Learning, 2011
- Evolutionary optimisation of noisy multi-objective problems using confidence-based dynamic resamplingEuropean Journal of Operational Research, 2010
- A multi-model selection framework for unknown and/or evolutive misclassification cost problemsPattern Recognition, 2010
- Modeling wine preferences by data mining from physicochemical propertiesDecision Support Systems, 2009
- Multi-class ROC analysis from a multi-objective optimisation perspectivePattern Recognition Letters, 2006
- Sequential kriging optimization using multiple-fidelity evaluationsStructural and Multidisciplinary Optimization, 2006
- Evolutionary Optimization in Uncertain Environments—A SurveyIEEE Transactions on Evolutionary Computation, 2005
- Performance assessment of multiobjective optimizers: an analysis and reviewIEEE Transactions on Evolutionary Computation, 2003
- A fast and elitist multiobjective genetic algorithm: NSGA-IIIEEE Transactions on Evolutionary Computation, 2002