DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6163-6173
Abstract
The rapid advances in Vision Transformer (ViT) refresh the state-of-the-art performances in various vision tasks, overshadowing the conventional CNN-based models. This ignites a few recent striking-back research in the CNN world showing that pure CNN models can achieve as good performance as ViT models when carefully tuned. While encouraging, designing such high-performance CNN models is challenging, requiring non-trivial prior knowledge of network design. To this end, a novel framework termed Mathematical Architecture Design for Deep CNN (Deep-MAD 1 1 Source codes are available at https://github.com/alibaba/lightweight-neural-architecture-search) is proposed to design high-performance CNN models in a principled way. In DeepMAD, a CNN network is modeled as an information processing system whose expressiveness and effectiveness can be analytically formulated by their structural parameters. Then a constrained mathematical programming (MP) problem is proposed to optimize these structural parameters. The MP problem can be easily solved by off-the-shelf MP solvers on CPUs with a small memory footprint. In addition, DeepMAD is a pure mathematical framework: no GPU or training data is required during network design. The superiority of DeepMAD is validated on multiple large-scale computer vision benchmark datasets. Notably on ImageNet-1k, only using conventional convolutional layers, DeepMAD achieves 0.7% and 1.5% higher top-1 accuracy than ConvNeXt and Swin on Tiny level, and 0.8% and 0.9% higher on Small level.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization2020
- Random Erasing Data AugmentationProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Regularized Evolution for Image Classifier Architecture SearchProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Mask R-CNNIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Squeeze-and-Excitation Networks2018
- Learning Transferable Architectures for Scalable Image Recognition2018
- Cascade R-CNN: Delving Into High Quality Object Detection2018
- Scene Parsing through ADE20K Dataset2017
- ImageNet: A large-scale hierarchical image database2009