Uncertainty-based quantization method for stable training of binary neural networks
Authors:
- Moscow Institute of Physics and Technology (National Research University)MUMoscow Institute of Physics and Technology (National Research University)
- A.V. TrusovATA.V. Trusov
- Federal Research Center “Computer Science and Control” of Russian Academy of SciencesFSFederal Research Center “Computer Science and Control” of Russian Academy of Sciences
- Llc “Smart Engines Service”LSLlc “Smart Engines Service”
- E.E. LimonovaELE.E. Limonova
- 1 August 2024
- journal article
- research article
- Published by Samara National Research University in Computer Optics
- Vol. 48 (4) , 573-581
Abstract
Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quantizer called UBQ (Uncertainty-based quantizer) for BNNs, which combines the advantages of existing methods, resulting in stable training and high-quality BNNs even with a low number of trainable parameters. We also propose a training method involving gradual network freezing and batch normalization replacement, facilitating a smooth transition from training mode to execution mode for BNNs. To evaluate UBQ, we conducted experiments on the MNIST and CIFAR-10 datasets and compared our method to existing algorithms. The results demonstrate that UBQ outperforms previous methods for smaller networks and achieves comparable results for larger networks.Keywords
This publication has 21 references indexed in Scilit:
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- ALODAD: An Anchor-Free Lightweight Object Detector for Autonomous DrivingIEEE Access, 2022
- An efficient GPU-accelerated inference engine for binary neural network on mobile phonesJournal of Systems Architecture, 2021
- AdderNet: Do We Really Need Multiplications in Deep Learning?2020
- Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks2019
- Quantization Networks2019
- Bi-Real Net: Enhancing the Performance of 1-Bit CNNs with Improved Representational Capability and Advanced Training Algorithm2018
- LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks2018
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference2018
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks2016