Compact Neural Network via Stacking Hybrid Units
Open Access
- 10 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (1) , 103-116
Abstract
As an effective tool for network compression, pruning techniques have been widely used to reduce the large number of parameters in deep neural networks (NNs). Nevertheless, unstructured pruning has the limitation of dealing with the sparse and irregular weights. By contrast, structured pruning can help eliminate this drawback but it requires complex criteria to determine which components to be pruned. Therefore, this paper presents a new method termed BUnit-Net, which directly constructs compact NNs by stacking designed basic units, without requiring additional judgement criteria anymore. Given the basic units of various architectures, they are combined and stacked systematically to build up compact NNs which involve fewer weight parameters due to the independence among the units. In this way, BUnit-Net can achieve the same compression effect as unstructured pruning while the weight tensors can still remain regular and dense. We formulate BUnit-Net in diverse popular backbones in comparison with the state-of-the-art pruning methods on different benchmark datasets. Moreover, two new metrics are proposed to evaluate the trade-off of compression performance. Experiment results show that BUnit-Net can achieve comparable classification accuracy while saving around 80% FLOPs and 73% parameters. That is, stacking basic units provides a new promising way for network compression.Keywords
All Related Versions
Funding Information
- NSFC and Research Grants Council (N_HKBU214/21)
- General Research Fund of RGC (12201321, 12202622)
- Hong Kong Baptist University (RC-FNRA-IG/18-19/SCI/03)
This publication has 60 references indexed in Scilit:
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks2016
- EIEACM SIGARCH Computer Architecture News, 2016
- Deep Residual Learning for Image Recognition2016
- Accelerating Very Deep Convolutional Networks for Classification and DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance PropagationPLOS ONE, 2015
- Fully convolutional networks for semantic segmentation2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Speeding up Convolutional Neural Networks with Low Rank Expansions2014
- Optimal Brain Surgeon and general network pruning2002
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998