NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search
- 1 March 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (11) , 8441-8455
Abstract
Neural architecture search (NAS) adopts a search strategy to explore the predefined search space to find superior architecture with the minimum searching costs. Bayesian optimization (BO) and evolutionary algorithms (EA) are two commonly used search strategies, but they suffer from being computationally expensive, challenging to implement, and exhibiting inefficient exploration ability. In this article, we propose a neural predictor guided EA to enhance the exploration ability of EA for NAS (NPENAS) and design two kinds of neural predictors. The first predictor is a BO acquisition function for which we design a graph-based uncertainty estimation network as the surrogate model. The second predictor is a graph-based neural network that directly predicts the performance of the input neural architecture. The NPENAS using the two neural predictors are denoted as NPENAS-BO and NPENAS-NP, respectively. In addition, we introduce a new random architecture sampling method to overcome the drawbacks of the existing sampling method. Experimental results on five NAS search spaces indicate that NPENAS-BO and NPENAS-NP outperform most existing NAS algorithms, with NPENAS-NP achieving state-of-the-art performance on four of the five search spaces.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (U19B2030, 61976167)
- Xi’an Science and Technology Program (201809170CX11JC12)
This publication has 24 references indexed in Scilit:
- Evolving Deep Convolutional Neural Networks for Image ClassificationIEEE Transactions on Evolutionary Computation, 2020
- Progressive Neural Architecture Search2018
- Learning Transferable Architectures for Scalable Image Recognition2018
- Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep ArchitectureIEEE Transactions on Neural Networks and Learning Systems, 2017
- Taking the Human Out of the Loop: A Review of Bayesian OptimizationProceedings of the IEEE, 2015
- Extreme Learning Machine for Multilayer PerceptronIEEE Transactions on Neural Networks and Learning Systems, 2015
- The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domainsIEEE Signal Processing Magazine, 2013
- OP-ELM: Optimally Pruned Extreme Learning MachineIEEE Transactions on Neural Networks, 2009
- Extreme learning machine: Theory and applicationsNeurocomputing, 2006
- Application of Bayesian approach to numerical methods of global and stochastic optimizationJournal of Global Optimization, 1994