AdaViT: Adaptive Tokens for Efficient Vision Transformer
Preprint
Abstract
We introduce A-ViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. A-ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds. We reformulate Adaptive Computation Time (ACT) for this task, extending halting to discard redundant spatial tokens. The appealing architectural properties of vision transformers enables our adaptive token reduction mechanism to speed up inference without modifying the network architecture or inference hardware. We demonstrate that A-ViT requires no extra parameters or sub-network for halting, as we base the learning of adaptive halting on the original network parameters. We further introduce distributional prior regularization that stabilizes training compared to prior ACT approaches. On the image classification task (ImageNet1K), we show that our proposed A-ViT yields high efficacy in filtering informative spatial features and cutting down on the overall compute. The proposed method improves the throughput of DeiT-Tiny by 62% and DeiT-Small by 38% with only 0.3% accuracy drop, outperforming prior art by a large margin. Project page at https://a-vit.github.io/Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions2021
- Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet2021
- UP-DETR: Unsupervised Pre-training for Object Detection with Transformers2021
- End-to-End Object Detection with Transformers2020
- Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference2020
- Resolution Adaptive Networks for Efficient Inference2020
- Convolutional Networks with Adaptive Inference GraphsInternational Journal of Computer Vision, 2019
- Dynamic Recursive Neural Network2019
- BlockDrop: Dynamic Inference Paths in Residual Networks2018
- Deep Neural Decision Forests2015