Good Performance Estimation Strategies are All You Need in Neural Architecture Search
- 1 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (3) , 2398-2412
Abstract
Recent advances in Neural Architecture Search (NAS) are essentially attributed to Performance Estimation (PE), i.e., a method aims to effectively estimate an architecture. Meanwhile, Kendall’s $\tau$ is well recognized as the principled evaluation criteria for PE strategies in the literature. We argue that Kendall’s $\tau$ is not the optimal solution. Through extensive experiments and theoretical analysis, we take the initiative to reveal the problem behind the Kendall’s $\tau$ and propose a novel criterion named Minimum Keeping Ratio (MKR), which is closely connected to the final performance of NAS. It allows us to compare different PE approaches in a unified perspective, and use effective ablation studies to verify common beliefs and key differences of PE strategies. Based on the findings from MKR, we are able to derive a simple NAS method by integrating different PE strategies with random sampling. Such a method shows very strong performance in efficiency and effectiveness through extensive experiments on different challenging benchmarks. In particular, our simple random sampling NAS finds the optimal architecture in NASbenchMacro, NASbench201, and NASbench301. It is also well generalized to different search spaces (MobileNet) and tasks (semantic segmentation), finding an architecture surpasses the previous state-of-the-art architectures by 4.25 mIoU under $ 600M$ FLOPs on ADE20K. Codes are available at https://anonymous.4open.science/r/Anonymization11264.
Keywords
Funding Information
- Shenzhen High-Level Talent Team Program
- Shenzhen Science and Technology Innovation Commission (KQTD 20240729102051063)
- National Science Fund for Distinguished Young Scholars (62025603)
- National Natural Science Foundation of China (62332002)
- Natural Science Foundation of Fujian Province of China (2021J06003)
- Fundamental Research Funds for the Central Universities
This publication has 47 references indexed in Scilit:
- EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search2021
- Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search2020
- BigNAS: Scaling up Neural Architecture Search with Big Single-Stage Models2020
- Single Path One-Shot Neural Architecture Search with Uniform Sampling2020
- Neural Predictor for Neural Architecture Search2020
- Neural Architecture Search2019
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- Progressive Neural Architecture Search2018
- Deep Residual Learning for Image Recognition2016
- ImageNet: A large-scale hierarchical image database2009