Network Sketching: Exploiting Binary Structure in Deep CNNs
Authors:
- Yiwen GuoYGYiwen Guo
- Anbang YaoAYAnbang Yao
- Hao ZhaoHZHao Zhao
- Yurong ChenYCYurong Chen
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 4040-4048
Abstract
Convolutional neural networks (CNNs) with deep architectures have substantially advanced the state-of-the-art in computer vision tasks. However, deep networks are typically resource-intensive and thus difficult to be deployed on mobile devices. Recently, CNNs with binary weights have shown compelling efficiency to the community, whereas the accuracy of such models is usually unsatisfactory in practice. In this paper, we introduce network sketching as a novel technique of pursuing binary-weight CNNs, targeting at more faithful inference and better trade-off for practical applications. Our basic idea is to exploit binary structure directly in pre-trained filter banks and to produce binary-weight models via tensor expansion. The whole process can be treated as a coarse-to-fine model approximation, akin to the pencil drawing steps of outlining and shading. To further speedup the generated models, namely the sketches, we also propose an associative implementation of binary tensor convolutions. Experimental results demonstrate that a proper sketch of AlexNet (or ResNet) outperforms the existing binary-weight models by large margins on the ImageNet large scale classification task, while the committed memory for network parameters only exceeds a little.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Deep Residual Learning for Image Recognition2016
- Quantized Convolutional Neural Networks for Mobile Devices2016
- Efficient and accurate approximations of nonlinear convolutional networks2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Data-free Parameter Pruning for Deep Neural Networks2015
- Integrated Imaging and Vision Techniques for Industrial Inspection2015
- Caffe2014
- Learning Separable Filters2013
- Adaptive greedy approximationsConstructive Approximation, 1997
- Shortest Connection Networks And Some GeneralizationsBell System Technical Journal, 1957