Regularizing Activation Distribution for Training Binarized Deep Networks
Preprint
Abstract
Binarized Neural Networks (BNNs) can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN's energy efficiency. In this paper, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Lightening the Load with Highly Accurate Storage- and Energy-Efficient LightNNsACM Transactions on Reconfigurable Technology and Systems, 2018
- Deep Learning with Low Precision by Half-Wave Gaussian Quantization2017
- An IoT Endpoint System-on-Chip for Secure and Energy-Efficient Near-Sensor AnalyticsIEEE Transactions on Circuits and Systems I: Regular Papers, 2017
- Exploring Normalization in Deep Residual Networks with Concatenated Rectified Linear UnitsProceedings of the AAAI Conference on Artificial Intelligence, 2017
- How to Train a Compact Binary Neural Network with High Accuracy?Proceedings of the AAAI Conference on Artificial Intelligence, 2017
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks2016
- Wide Residual Networks2016