Fast and Accurate Model Scaling
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 924-932
Abstract
In this work we analyze strategies for convolutional neural network scaling; that is, the process of scaling a base convolutional network to endow it with greater computational complexity and consequently representational power. Example scaling strategies may include increasing model width, depth, resolution, etc. While various scaling strategies exist, their tradeoffs are not fully understood. Existing analysis typically focuses on the interplay of accuracy and flops (floating point operations). Yet, as we demonstrate, various scaling strategies affect model parameters, activations, and consequently actual runtime quite differently. In our experiments we show the surprising result that numerous scaling strategies yield networks with similar accuracy but with widely varying properties. This leads us to propose a simple fast compound scaling strategy that encourages primarily scaling model width, while scaling depth and resolution to a lesser extent. Unlike currently popular scaling strategies, which result in about O(s) increase in model activation w.r.t. scaling flops by a factor of s, the proposed fast compound scaling results in close to $O\left( {\sqrt s } \right)$ increase in activations, while achieving excellent accuracy. Fewer activations leads to speedups on modern memory-bandwidth limited hardware (e.g., GPUs). More generally, we hope this work provides a framework for analyzing scaling strategies under various computational constraints.
Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Squeeze-and-Excitation Networks2018
- Learning Transferable Architectures for Scalable Image Recognition2018
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era2017
- Xception: Deep Learning with Depthwise Separable Convolutions2017
- Aggregated Residual Transformations for Deep Neural Networks2017
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Going deeper with convolutions2015
- ImageNet: A large-scale hierarchical image database2009