SiGeo: Sub-One-Shot NAS via Geometry of Loss Landscape
Open Access
- 24 August 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4536-4547
Abstract
Neural Architecture Search (NAS) has become a widely used tool for automating neural network design. While one-shot NAS methods have successfully reduced computational requirements, they often require extensive training. On the other hand, zero-shot NAS utilizes training-free proxies to evaluate a candidate architecture's test performance but has two limitations: (1) inability to use the information gained as a network improves with training and (2) unreliable performance, particularly in complex domains like RecSys, due to the multi-modal data inputs and complex architecture configurations. To synthesize the benefits of both methods, we introduce a "sub-one-shot" paradigm that serves as a bridge between zero-shot and one-shot NAS. In sub-one-shot NAS, the supernet is trained using only a small subset of the training data, a phase we refer to as "warm-up." Within this framework, we present SiGeo, a proxy founded on a novel theoretical framework that connects the supernet warm-up with the efficacy of the proxy. Extensive experiments have consistently shown that SiGeo, when properly warmed up, surpasses state-of-the-art NAS proxies in many established NAS benchmarks in the computer vision domain. Furthermore, when tested on recommendation system benchmarks, SiGeo demonstrates its ability to match the performance of state-of-the-art weight-sharing one-shot NAS methods while significantly reducing computational costs by approximately 60%.Keywords
This publication has 16 references indexed in Scilit:
- NPENAS: Neural Predictor Guided Evolution for Neural Architecture SearchIEEE Transactions on Neural Networks and Learning Systems, 2022
- Zen-NAS: A Zero-Shot NAS for High-Performance Image Recognition2021
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search2021
- EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search2021
- TransNAS-Bench-101: Improving transferability and Generalizability of Cross-Task Neural Architecture Search2021
- Neural Predictor for Neural Architecture Search2020
- Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation2020
- Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation2019
- Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance PredictorIEEE Transactions on Evolutionary Computation, 2019
- Searching for a Robust Neural Architecture in Four GPU Hours2019