Differential Evolution Based Layer-Wise Weight Pruning for Compressing Deep Neural Networks
Open Access
Abstract
Deep neural networks have evolved significantly in the past decades and are now able to achieve better progression of sensor data. Nonetheless, most of the deep models verify the ruling maxim in deep learning—bigger is better—so they have very complex structures. As the models become more complex, the computational complexity and resource consumption of these deep models are increasing significantly, making them difficult to perform on resource-limited platforms, such as sensor platforms. In this paper, we observe that different layers often have different pruning requirements, and propose a differential evolutionary layer-wise weight pruning method. Firstly, the pruning sensitivity of each layer is analyzed, and then the network is compressed by iterating the weight pruning process. Unlike some other methods that deal with pruning ratio by greedy ways or statistical analysis, we establish an optimization model to find the optimal pruning sensitivity set for each layer. Differential evolution is an effective method based on population optimization which can be used to address this task. Furthermore, we adopt a strategy to recovery some of the removed connections to increase the capacity of the pruned model during the fine-tuning phase. The effectiveness of our method has been demonstrated in experimental studies. Our method compresses the number of weight parameters in LeNet-300-100, LeNet-5, AlexNet and VGG16 by , , and , respectively.
Keywords
This publication has 15 references indexed in Scilit:
- A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning2017
- Structured Pruning of Deep Convolutional Neural NetworksACM Journal on Emerging Technologies in Computing Systems, 2017
- Deep Residual Learning for Image Recognition2016
- PRUNING ARTIFICIAL NEURAL NETWORKS USING NEURAL COMPLEXITY MEASURESInternational Journal of Neural Systems, 2008
- Differential Evolution Algorithm With Strategy Adaptation for Global Numerical OptimizationIEEE Transactions on Evolutionary Computation, 2008
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous SpacesJournal of Global Optimization, 1997
- Pruning backpropagation neural networks using modern stochastic optimisation techniquesNeural Computing and Applications, 1997
- An iterative pruning algorithm for feedforward neural networksIEEE Transactions on Neural Networks, 1997
- Pruning algorithms-a surveyIEEE Transactions on Neural Networks, 1993