Discriminative Layer Pruning for Convolutional Neural Networks
- 24 February 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 14 (4) , 828-837
Abstract
The predictive ability of convolutional neural networks (CNNs) can be improved by increasing their depth. However, increasing depth also increases computational cost significantly, in terms of both floating point operations and memory consumption, hindering applicability on resource-constrained systems such as mobile and internet of things (IoT) devices. Fortunately, most networks have spare capacity, that is, they require fewer parameters than they actually have to perform accurately. This motivates network compression methods, which remove or quantize parameters to improve resource-efficiency. In this work, we consider a straightforward strategy for removing entire convolutional layers to reduce network depth. Since it focuses on depth, this approach not only reduces memory usage, but also reduces prediction time significantly by mitigating the serialization overhead incurred by forwarding through consecutive layers. We show that a simple subspace projection approach can be employed to estimate the importance of network layers, enabling the pruning of CNNs to a resource-efficient depth within a given network size constraint. We estimate importance on a subspace computed using Partial Least Squares, a feature projection approach that preserves discriminative information. Consequently, this importance estimation is correlated to the contribution of the layer to the classification ability of the model. We show that cascading discriminative layer pruning with filter-oriented pruning improves the resource-efficiency of the resulting network compared to using any of them alone, and that it outperforms state-of-the-art methods. Moreover, we show that discriminative layer pruning alone, without cascading, achieves competitive resource-efficiency compared to methods that prune filters from all layers.Keywords
Funding Information
- Brazilian National Research Council – CNPq (#311053/2016-5, #438629/2018-3)
- Minas Gerais Research Foundation – FAPEMIG (APQ-00567-14, PPM-00540-17)
- Coordination for the Improvement of Higher Education Personnel – CAPES
This publication has 27 references indexed in Scilit:
- Structured Pruning of Deep Convolutional Neural NetworksACM Journal on Emerging Technologies in Computing Systems, 2017
- Deep Residual Learning for Image Recognition2016
- Infinite Feature Selection2015
- Going deeper with convolutions2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Partial least squares regression on grassmannian manifold for emotion recognition2013
- A review of variable selection methods in Partial Least Squares RegressionChemometrics and Intelligent Laboratory Systems, 2012
- Simultaneous dimensionality reduction and human age estimation via kernel partial least squares regression2011
- Neuroevolution: from architectures to learningEvolutionary Intelligence, 2008
- PCA and PLS with very large data setsComputational Statistics & Data Analysis, 2005