Effect of the Sampling of a Dataset in the Hyperparameter Optimization Phase over the Efficiency of a Machine Learning Algorithm
Open Access
- 1 January 2019
- journal article
- research article
- Published by Wiley in Complexity
- Vol. 2019 (1)
Abstract
Selecting the best configuration of hyperparameter values for a Machine Learning model yields directly in the performance of the model on the dataset. It is a laborious task that usually requires deep knowledge of the hyperparameter optimizations methods and the Machine Learning algorithms. Although there exist several automatic optimization techniques, these usually take significant resources, increasing the dynamic complexity in order to obtain a great accuracy. Since one of the most critical aspects in this computational consume is the available dataset, among others, in this paper we perform a study of the effect of using different partitions of a dataset in the hyperparameter optimization phase over the efficiency of a Machine Learning algorithm. Nonparametric inference has been used to measure the rate of different behaviors of the accuracy, time, and spatial complexity that are obtained among the partitions and the whole dataset. Also, a level of gain is assigned to each partition allowing us to study patterns and allocate whose samples are more profitable. Since Cybersecurity is a discipline in which the efficiency of Artificial Intelligence techniques is a key aspect in order to extract actionable knowledge, the statistical analyses have been carried out over five Cybersecurity datasets.Keywords
Funding Information
- European Regional Development Fund
This publication has 32 references indexed in Scilit:
- Intelligent rule‐based phishing websites classificationIET Information Security, 2014
- Exploring high-level features for detecting cyberpedophiliaComputer Speech & Language, 2014
- Predicting phishing websites based on self-structuring neural networkNeural Computing and Applications, 2013
- Sequential Model-Based Optimization for General Algorithm Configuration2011
- Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of powerInformation Sciences, 2010
- The particle swarm - explosion, stability, and convergence in a multidimensional complex spaceIEEE Transactions on Evolutionary Computation, 2002
- Random ForestsMachine Learning, 2001
- Completely Derandomized Self-Adaptation in Evolution StrategiesEvolutionary Computation, 2001
- Gradient-Based Optimization of HyperparametersNeural Computation, 2000
- A Simplex Method for Function MinimizationThe Computer Journal, 1965