Layer Pruning for Obtaining Shallower ResNets
Authors:
- 28 April 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 29 (10709908) , 1172-1176
Abstract
Network pruning has become an effective scheme to cut down the network complexity and speed up the inference. Current work mainly focuses on filter pruning, which deletes filters in a structured way. Although the calculation amount is reduced after pruning, the acceleration effect is not obvious. In this letter, we present a layer pruning method to obtain shallower ResNets (LPSR) and significantly reduce the inference latency. Taking advantage of the characteristics of the residual module, we disconnect the unimportant residual mapping and reserve only the identity mapping, thereby equivalently removing the corresponding residual module and achieving the purpose of layer pruning. We adopt Taylor expansion to estimate the impact of disconnecting BN layers in the residual mapping. These impacts are then normalized and used to identify unimportant residual blocks to prune. Comprehensive experimental results demonstrate that LPSR not only reduces the number of parameters and calculations, but also significantly speed up the inference. LPSR improves the accuracy of ResNet-56 on CIFAR-10 from 93.21% to 93.40%, while reducing 44.71% parameters, 52.68% calculations and 44.89% inference latency.Keywords
This publication has 19 references indexed in Scilit:
- Shallowing Deep Networks: Layer-Wise Pruning Based on Feature RepresentationsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration2019
- Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks2018
- NISP: Pruning Networks Using Neuron Importance Score Propagation2018
- Recent advances in efficient computation of deep convolutional neural networksFrontiers of Information Technology & Electronic Engineering, 2018
- ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression2017
- Learning Efficient Convolutional Networks through Network Slimming2017
- More is Less: A More Complicated Network with Less Inference Complexity2017
- Deep Residual Learning for Image Recognition2016
- ImageNet: A large-scale hierarchical image database2009