Multi-Objective Hyperparameter Tuning and Feature Selection using Filter Ensembles
Preprint
Abstract
Both feature selection and hyperparameter tuning are key tasks in machine learning. Hyperparameter tuning is often useful to increase model performance, while feature selection is undertaken to attain sparse models. Sparsity may yield better model interpretability and lower cost of data acquisition, data handling and model inference. While sparsity may have a beneficial or detrimental effect on predictive performance, a small drop in performance may be acceptable in return for a substantial gain in sparseness. We therefore treat feature selection as a multi-objective optimization task. We perform hyperparameter tuning and feature selection simultaneously because the choice of features of a model may influence what hyperparameters perform well. We present, benchmark, and compare two different approaches for multi-objective joint hyperparameter optimization and feature selection: The first uses multi-objective model-based optimization. The second is an evolutionary NSGA-II-based wrapper approach to feature selection which incorporates specialized sampling, mutation and recombination operators. Both methods make use of parameterized filter ensembles. While model-based optimization needs fewer objective evaluations to achieve good performance, it incurs computational overhead compared to the NSGA-II, so the preferred choice depends on the cost of evaluating a model on given data.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Ensembles for feature selection: A review and future trendsInformation Fusion, 2019
- Auto-sklearn: Efficient and Robust Automated Machine Learning2019
- A multi-objective genetic algorithm for simultaneous model and feature selection for support vector machinesArtificial Intelligence Review, 2017
- Model-Based Multi-objective Optimization: Taxonomy, Multi-Point Proposal, Toolbox and Benchmark2015
- Multi-Objective Feature Subset Selection using Non-dominated Sorting Genetic AlgorithmJournal of Applied Research and Technology, 2015
- Comparing Two New Gene Selection Ensemble Approaches with the Commonly-Used Approach2012
- A fast and elitist multiobjective genetic algorithm: NSGA-IIIEEE Transactions on Evolutionary Computation, 2002
- Kernel Nearest-Neighbor AlgorithmNeural Processing Letters, 2002
- Efficient Global Optimization of Expensive Black-Box FunctionsJournal of Global Optimization, 1998
- Wrappers for feature subset selectionArtificial Intelligence, 1997