SFAO: Sign-Flipping-Aware Optimization for Early-Stopping of Binarized Neural Networks
Open Access
- 13 November 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 128306-128315
Abstract
One of the vital challenges for the binary neural networks (BNNs) is improving their inference performance by expanding their data representation capabilities for figuring out delicate patterns and nuances in the data. Addressing the explosive computational demands on neural network training is essential to guarantee sustainable development and scalable deployment. However, mitigating the increase in the computational cost during the training phase is critical for ensuring sustainability and scalability during deployment. In this study, an advanced sign-flipping-aware optimizer (SFAO) that focuses on BNNs was introduced to diminish the computational burden. SFAO balanced the model performance and computational cost through sign-flipping-aware updating rules throughout the training of BNNs. SFAO optimizer, tailored for BNNs with binary weight-specific updating rules, considerably reduced the computing resources needed for training on the CIFAR-10 dataset. Specifically, it surpassed the conventional full-precision updating rule by reducing the total instruction count by 21.89%. In contrast, SFAO showed a marginal 0.44% decline in the image classification accuracy relative to the updating rules for the full-precision parameters. Furthermore, the implementation of early stopping using the sign flip rate led to a notable reduction of 9.37% in the average computation time per network for the ImageNet dataset.Keywords
Funding Information
- BK21 FOUR project
- Institude of Information & Communication Technology Planning & Evaluation (2019-0-00421)
- the Ministry of Trade, Industry & Energy (1415187686)
- Samsung Electronics Company Ltd. (IO201209-07877-01)
- National Research Foundation of Korea (2020M3H2A1076786)
This publication has 28 references indexed in Scilit:
- BitStream2018
- Bi-Real Net: Enhancing the Performance of 1-Bit CNNs with Improved Representational Capability and Advanced Training Algorithm2018
- LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks2018
- Training Binary Weight Networks via Semi-Binary Decomposition2018
- Adaptive weighted nonnegative low-rank representationPattern Recognition, 2018
- Two-Step Quantization for Low-bit Neural Networks2018
- Channel Pruning for Accelerating Very Deep Neural Networks2017
- Compressing deep neural networks for efficient visual inference2017
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks2016
- The gem5 simulatorACM SIGARCH Computer Architecture News, 2011