Low-bit Quantization of Neural Networks for Efficient Inference
Authors:
Top Cited Papers
- 1 October 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3009-3018
Abstract
Recent machine learning methods use increasingly large deep neural networks to achieve state of the art results in various tasks. The gains in performance come at the cost of a substantial increase in computation and storage requirements. This makes real-time implementations on limited resources hardware a challenging task. One popular approach to address this challenge is to perform low-bit precision computations via neural network quantization. However, aggressive quantization generally entails a severe penalty in terms of accuracy, and often requires retraining of the network, or resorting to higher bit precision quantization. In this paper, we formalize the linear quantization task as a Minimum Mean Squared Error (MMSE) problem for both weights and activations, allowing low-bit precision inference without the need for full network retraining. We propose the analysis and the optimization of constrained MSE problems for performant hardware aware quantization. The proposed approach allows 4 bits integer (INT4) quantization for deployment of pretrained models on limited hardware resources. Multiple experiments on various network architectures show that the suggested method yields state of the art results with minimal loss of tasks accuracy.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- SYQ: Learning Symmetric Quantization for Efficient Deep Neural Networks2018
- Performance Guaranteed Network Acceleration via High-Order Residual Quantization2017
- Densely Connected Convolutional Networks2017
- In-Datacenter Performance Analysis of a Tensor Processing Unit2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Fixed-point feedforward deep neural network design using weights +1, 0, and −12014
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012