ECQ$^{\text{x}}$: Explainability-Driven Quantization for Low-Bit and Sparse DNNs
Preprint
Abstract
The remarkable success of deep neural networks (DNNs) in various applications is accompanied by a significant increase in network parameters and arithmetic operations. Such increases in memory and computational demands make deep learning prohibitive for resource-constrained hardware platforms such as mobile devices. Recent efforts aim to reduce these overheads, while preserving model performance as much as possible, and include parameter reduction techniques, parameter quantization, and lossless compression techniques. In this chapter, we develop and describe a novel quantization paradigm for DNNs: Our method leverages concepts of explainable AI (XAI) and concepts of information theory: Instead of assigning weight values based on their distances to the quantization clusters, the assignment function additionally considers weight relevances obtained from Layer-wise Relevance Propagation (LRP) and the information content of the clusters (entropy optimization). The ultimate goal is to preserve the most relevant weights in quantization clusters of highest information content. Experimental results show that this novel Entropy-Constrained and XAI-adjusted Quantization (ECQ$^{\text{x}}$) method generates ultra low-precision (2-5 bit) and simultaneously sparse neural networks while maintaining or even improving model performance. Due to reduced parameter precision and high number of zero-elements, the rendered networks are highly compressible in terms of file size, up to $103\times$ compared to the full-precision unquantized DNN model. Our approach was evaluated on different types of models and datasets (including Google Speech Commands, CIFAR-10 and Pascal VOC) and compared with previous work.
Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Pruning by explaining: A novel criterion for deep neural network pruningPattern Recognition, 2021
- FantastIC4: A Hardware-Software Co-Design Approach for Efficiently Running 4Bit-Compact Multilayer PerceptronsIEEE Open Journal of Circuits and Systems, 2021
- Explaining Deep Neural Networks and Beyond: A Review of Methods and ApplicationsProceedings of the IEEE, 2021
- Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive SurveyProceedings of the IEEE, 2020
- DeepCABAC: A Universal Compression Algorithm for Deep Neural NetworksIEEE Journal of Selected Topics in Signal Processing, 2020
- Improved protein structure prediction using potentials from deep learningNature, 2020
- Methods for interpreting and understanding deep neural networksDigital Signal Processing, 2018
- Efficient Processing of Deep Neural Networks: A Tutorial and SurveyProceedings of the IEEE, 2017
- Quantum-chemical insights from deep tensor neural networksNature Communications, 2017
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009