On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
Authors:
Preprint
Abstract
Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.Keywords
All Related Versions
This publication has 56 references indexed in Scilit:
- TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning2019
- A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoringExpert Systems with Applications, 2017
- A Novel Hardware/Software Partitioning Method Based on Position Disturbed Particle Swarm Optimization with Invasive Weed OptimizationJournal of Computer Science and Technology, 2017
- A review of automatic selection methods for machine learning algorithms and hyper-parameter valuesNetwork Modeling Analysis in Health Informatics and Bioinformatics, 2016
- Applied Predictive Modeling2013
- Fast and Accurate Sentiment Classification Using an Enhanced Naive Bayes Model2013
- Large-Scale Machine Learning with Stochastic Gradient Descent2010
- On kernel difference-weighted k-nearest neighbor classificationPattern Analysis and Applications, 2008
- A novel population initialization method for accelerating evolutionary algorithmsComputers & Mathematics with Applications, 2007
- Comparing Fuzzy Naive Bayes and Gaussian Naive Bayes for Decision Making in RoboCup 3D2006