SMASH: One-Shot Model Architecture Search through HyperNetworks
Preprint
Abstract
Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. Our code is available at https://github.com/ajbrock/SMASHKeywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Deep Networks with Stochastic Depth2016
- Deep Residual Learning for Image Recognition2016
- Wide Residual Networks2016
- 3D ShapeNets: A deep representation for volumetric shapes2015
- A Hypercube-Based Encoding for Evolving Large-Scale Neural NetworksArtificial Life, 2009
- Modeling systems with internal state using evolino2005