BinaryFormer: A Hierarchical-Adaptive Binary Vision Transformer (ViT) for Efficient Computing
- 15 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 20 (8) , 10657-10668
Abstract
Vision Transformer (ViT) has recently demonstrated impressive nonlinear modeling capabilities and achieved state-of-the-art performance in various industrial applications, such as object recognition, anomaly detection, and robot control. However, their practical deployment can be hindered by high storage requirements and computational intensity. To alleviate these challenges, we propose a binary transformer called BinaryFormer, which quantizes the learned weights of the ViT module from 32-b precision to 1 b. Furthermore, we propose a hierarchical-adaptive architecture that replaces expensive matrix operations with more affordable addition and bit operations by switching between two attention modes. As a result, BinaryFormer is able to effectively compress the model size as well as reduce the computation cost of ViT. Experimental results on the ImageNet-1K benchmark datasets show that BinaryFormer reduces the size of a typical ViT model by an average of 27.7× and converts over 99% of multiplication operations into bit operations while maintaining reasonable accuracy.Keywords
Funding Information
- National Natural Science Foundation of China (61701310, 62372306)
- Natural Science Foundation of Guangdong Province (2024A1515011972, 2023A1515011197, 2022A1515011245)
- Natural Science Foundation of Shenzhen Municipality (20220809160139001)
This publication has 30 references indexed in Scilit:
- Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- ReCU: Reviving the Dead Weights in Binary Neural Networks2021
- Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions2021
- Low-bit Quantization of Neural Networks for Efficient Inference2019
- Unified Perceptual Parsing for Scene Understanding2018
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference2018
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks2016
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009