Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
Authors:
- 1 August 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 109-116
Abstract
Vision transformers (ViTs) are emerging with significantly improved accuracy in computer vision tasks. However, their complex architecture and enormous computation/storage demand impose urgent needs for new hardware accelerator design methodology. This work proposes an FPGA-aware automatic ViT acceleration framework based on the proposed mixed-scheme quantization. To the best of our knowledge, this is the first FPGA-based ViT acceleration framework exploring model quantization. Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0.47% to 1.36% higher Top-l accuracy under the same bit-width. Compared with the 32-bit floating-point baseline FPGA accelerator, our accelerator achieves around 5.6x improvement on the frame rate (i.e., 56.8 FPS vs. 10.0 FPS) with 0.71% accuracy drop on ImageNet dataset for DeiT-base.Keywords
Funding Information
- NSF (CCF-1901378)
This publication has 16 references indexed in Scilit:
- Accommodating Transformer onto FPGA2021
- Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework2021
- FTRANS2020
- TernaryBERT: Distillation-aware Ultra-low Bit BERT2020
- HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision2019
- HAQ: Hardware-Aware Automated Quantization With Mixed Precision2019
- Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network Using Truncated Gaussian Approximation2019
- Aggregated Residual Transformations for Deep Neural Networks2017
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks2016
- Deep Residual Learning for Image Recognition2016