Uncertainty-based quantization method for stable training of binary neural networks

Authors:
  • MUMoscow Institute of Physics and Technology (National Research University)
    Moscow Institute of Physics and Technology (National Research University)
  • FSFederal Research Center “Computer Science and Control” of Russian Academy of Sciences
    Federal Research Center “Computer Science and Control” of Russian Academy of Sciences
  • LSLlc “Smart Engines Service”
    Llc “Smart Engines Service”

Abstract
Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quantizer called UBQ (Uncertainty-based quantizer) for BNNs, which combines the advantages of existing methods, resulting in stable training and high-quality BNNs even with a low number of trainable parameters. We also propose a training method involving gradual network freezing and batch normalization replacement, facilitating a smooth transition from training mode to execution mode for BNNs. To evaluate UBQ, we conducted experiments on the MNIST and CIFAR-10 datasets and compared our method to existing algorithms. The results demonstrate that UBQ outperforms previous methods for smaller networks and achieves comparable results for larger networks.