Binary Transformer Based on the Alignment and Correction of Distribution
Open Access
Abstract
Transformer is a powerful model widely used in artificial intelligence applications. It contains complex structures and has extremely high computational requirements that are not suitable for embedded intelligent sensors with limited computational resources. The binary quantization technology takes up less memory space and has a faster calculation speed; however, it is seldom studied for the lightweight transformer. Compared with full-precision networks, the key bottleneck lies in the distribution shift problem caused by the existing binary quantization methods. To tackle this problem, the feature distribution alignment operation in binarization is investigated. The median shift and mean restore is designed to ensure consistency between the binary feature distribution and the full-precision transformer. Then, a knowledge distillation architecture for distribution correction is developed, which has a teacher–student structure comprising a full-precision and binary transformer, to further rectify the feature distribution of the binary student network to ensure the completeness and accuracy of the data. Experimental results on the CIFAR10, CIFAR100, ImageNet-1k, and TinyImageNet datasets show the effectiveness of the proposed binary optimization model, which outperforms the previous state-of-the-art binarization mechanisms while maintaining the same computational complexity.Keywords
Funding Information
- Yunnan Outstanding Young Talent Project (202301AV070003, 202302AG050009, 202202AD080013, YYZN-2024-1, ghfund202407010460)
- Yunnan Major Science and Technology Project (202301AV070003, 202302AG050009, 202202AD080013, YYZN-2024-1, ghfund202407010460)
- Special Education language Intelligence Sichuan key Laboratory of philosophy and Social Sciences Open Project (202301AV070003, 202302AG050009, 202202AD080013, YYZN-2024-1, ghfund202407010460)
- photosynthetic Fund (202301AV070003, 202302AG050009, 202202AD080013, YYZN-2024-1, ghfund202407010460)
This publication has 23 references indexed in Scilit:
- BinaryFormer: A Hierarchical-Adaptive Binary Vision Transformer (ViT) for Efficient ComputingIEEE Transactions on Industrial Informatics, 2024
- Universal Binary Neural Networks Design by Improved Differentiable Neural Architecture SearchIEEE Transactions on Circuits and Systems for Video Technology, 2024
- Self-knowledge distillation enhanced binary neural networks derived from underutilized informationApplied Intelligence, 2024
- A comprehensive review of Binary Neural NetworkArtificial Intelligence Review, 2023
- Distribution-Sensitive Information Retention for Accurate Binary Neural NetworkInternational Journal of Computer Vision, 2022
- RB-Net: Training Highly Accurate and Efficient Binary Neural Networks With Reshaped Point-Wise Convolution and Balanced ActivationIEEE Transactions on Circuits and Systems for Video Technology, 2022
- Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers2021
- Deep learning in multi-object detection and tracking: state of the artApplied Intelligence, 2021
- The classification and denoising of image noise based on deep neural networksApplied Intelligence, 2020
- ImageNet: A large-scale hierarchical image database2009