MNGNAS: Distilling Adaptive Combination of Multiple Searched Networks for One-Shot Neural Architecture Search

Authors:
Publisher website
Google Scholar
Abstract
Recently neural architecture (NAS) search has attracted great interest in academia and industry. It remains a challenging problem due to the huge search space and computational costs. Recent studies in NAS mainly focused on the usage of weight sharing to train a SuperNet once. However, the corresponding branch of each subnetwork is not guaranteed to be fully trained. It may not only incur huge computation costs but also affect the architecture ranking in the retraining procedure. We propose a multi-teacher-guided NAS, which proposes to use the adaptive ensemble and perturbation-aware knowledge distillation algorithm in the one-shot-based NAS algorithm. The optimization method aiming to find the optimal descent directions is used to obtain adaptive coefficients for the feature maps of the combined teacher model. Besides, we propose a specific knowledge distillation process for optimal architectures and perturbed ones in each searching process to learn better feature maps for later distillation procedures. Comprehensive experiments verify our approach is flexible and effective. We show improvement in precision and search efficiency in the standard recognition dataset. We also show improvement in correlation between the accuracy of the search algorithm and true accuracy by NAS benchmark datasets.
Funding Information
  • National Natural Science Foundation of China (62272164, 62272298, 62077037)
  • National Key Research and Development Program of China (2022YFC2407000)
  • Interdisciplinary Program of Shanghai Jiao Tong University (YG2023LC11, YG2022ZD007)
  • College-level Project Fund of Shanghai Jiao Tong University Affiliated Sixth People's Hospital (ynlc201909)
  • Medical-industrial Cross-fund of Shanghai Jiao Tong University (YG2022QN089)
  • Science and Technology on Space Intelligent Control Laboratory (HTKJ2022KL502010)
  • Hong Kong Polytechnic University (P0042740, P0030419, P0043906, P0044520)

This publication has 32 references indexed in Scilit: