An Investigation on Hardware-Aware Vision Transformer Scaling
Open Access
- 11 May 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Embedded Computing Systems
- Vol. 23 (3) , 1-19
Abstract
Vision Transformer (ViT) has demonstrated promising performance in various computer vision tasks, and recently attracted a lot of research attention. Many recent works have focused on proposing new architectures to improve ViT and deploying it into real-world applications. However, little effort has been made to analyze and understand ViT’s architecture design space and its implication for hardware costs on different devices. In this work, by simply scaling ViT’s depth, width, input size, and other basic configurations, we show that a scaled vanilla ViT model without bells and whistles can achieve comparable or superior accuracy-efficiency trade-off than most of the latest ViT variants. Specifically, compared with DeiT-Tiny, our scaled model achieves a ↑ 1.9% higher ImageNet top-1 accuracy under the same FLOPs and a ↑ 3.7% better ImageNet top-1 accuracy under the same latency on an NVIDIA Edge GPU TX2. Motivated by this, we further investigate the extracted scaling strategies from the following two aspects: (1) can these scaling strategies be transferred across different real hardware devices ? and (2) can these scaling strategies be transferred to different ViT variants and tasks ?. For (1), our exploration, based on various devices with different resource budgets, indicates that the transferability effectiveness depends on the underlying device together with its corresponding deployment tool. For (2), we validate the effective transferability of the aforementioned scaling strategies obtained from a vanilla ViT model on top of an image classification task to the PiT model, a strong ViT variant targeting efficiency as well as object detection and video classification tasks. In particular, when transferred to PiT, our scaling strategies lead to a boosted ImageNet top-1 accuracy of from 74.6% to 76.7% (↑ 2.1%) under the same 0.7G FLOPs. When transferred to the COCO object detection task, the average precision is boosted by ↑ 0.7% under a similar throughput on a V100 GPU.Keywords
Funding Information
- NSF CCRI (2016727)
- NSF RTML (1937592)
This publication has 18 references indexed in Scilit:
- Searching for MobileNetV32019
- Selective Kernel Networks2019
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Squeeze-and-Excitation Networks2018
- A Comparative Study of Real-Time Semantic Segmentation for Autonomous Driving2018
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era2017
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014
- ImageNet: A large-scale hierarchical image database2009
- Feature selection: evaluation, application, and small sample performanceIEEE Transactions on Pattern Analysis and Machine Intelligence, 1997