AdaBits: Neural Network Quantization With Adaptive Bit-Widths
Authors:
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2143-2153
Abstract
Deep neural networks with adaptive configurations have gained increasing attention due to the instant and flexible deployment of these models on platforms with different resource budgets. In this paper, we investigate a novel option to achieve this goal by enabling adaptive bit-widths of weights and activations in the model. We first examine the benefits and challenges of training quantized model with adaptive bit-widths, and then experiment with several approaches including direct adaptation, progressive training and joint training. We discover that joint training is able to produce comparable performance on the adaptive model as individual models. We also propose a new technique named Switchable Clipping Level (S-CL) to further improve quantized models at the lowest bit-width. With our proposed techniques applied on a bunch of models including MobileNet V1/V2 and ResNet50, we demonstrate that bit-width of weights and activations is a new option for adaptively executable deep neural networks, offering a distinct opportunity for improved accuracy-efficiency trade-off as well as instant adaptation according to the platform constraints in real-world applications.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- A survey of deep learning techniques for autonomous drivingJournal of Field Robotics, 2019
- Hyper-Pairing Network for Multi-phase Pancreatic Ductal Adenocarcinoma Segmentation2019
- Direct Shape Regression Networks for End-to-End Face Alignment2018
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Learning Transferable Architectures for Scalable Image Recognition2018
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference2018
- Balanced Quantization: An Effective and Efficient Approach to Quantized Neural NetworksJournal of Computer Science and Technology, 2017
- Weighted-Entropy-Based Quantization for Deep Neural Networks2017
- Deep Residual Learning for Image Recognition2016
- Practical object recognition in autonomous driving and beyond2011