Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss
Authors:
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4345-4354
Abstract
Reducing bit-widths of activations and weights of deep networks makes it efficient to compute and store them in memory, which is crucial in their deployments to resource-limited devices, such as mobile phones. However, decreasing bit-widths with quantization generally yields drastically degraded accuracy. To tackle this problem, we propose to learn to quantize activations and weights via a trainable quantizer that transforms and discretizes them. Specifically, we parameterize the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network. This quantization-interval-learning (QIL) allows the quantized networks to maintain the accuracy of the full-precision (32-bit) networks with bit-width as low as 4-bit and minimize the accuracy degeneration with further bit-width reduction (i.e., 3 and 2-bit). Moreover, our quantizer can be trained on a heterogeneous dataset, and thus can be used to quantize pretrained networks without access to their training data. We demonstrate the effectiveness of our trainable quantizer on ImageNet dataset with various network architectures such as ResNet-18, -34 and AlexNet, on which it outperforms existing methods to achieve the state-of-the-art accuracy.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks2018
- Towards Effective Low-Bitwidth Convolutional Neural Networks2018
- SYQ: Learning Symmetric Quantization for Efficient Deep Neural Networks2018
- Two-Step Quantization for Low-bit Neural Networks2018
- Balanced Quantization: An Effective and Efficient Approach to Quantized Neural NetworksJournal of Computer Science and Technology, 2017
- Weighted-Entropy-Based Quantization for Deep Neural Networks2017
- Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network2017
- Deep Learning with Low Precision by Half-Wave Gaussian Quantization2017
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015