Optimizing Vision Transformers: Unveiling ‘Focus and Forget’ for Enhanced Computational Efficiency
Open Access
- 10 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 27908-27927
Abstract
Vision Transformers are renowned for their accuracy in computer vision tasks but are computationally and memory expensive, making them challenging to deploy on resource-constrained edge devices. In our research paper, we introduce a revolutionary approach to designing energy-aware dynamically prunable Vision Transformers for use in edge applications. Our solution denoted as Incremental Resolution Enhancing Transformer (IRET), works by the sequential sampling of the input image. However, in our case, the embedding size of input tokens is considerably smaller than prior-art solutions. This embedding is used in the first few layers of the IRET vision transformer until a reliable attention matrix is formed. Then the attention matrix is used to sample additional information using a learnable 2D lifting scheme only for important tokens and IRET drops the tokens receiving low attention scores. Hence, as the model pays more attention to a subset of tokens for its task, its focus and resolution also increase. This incremental attention-guided sampling of input and dropping of unattended tokens allow IRET to significantly prune its computation tree on demand. By controlling the threshold for dropping unattended tokens and increasing the focus of attended ones, we can train a model that dynamically trades off complexity for accuracy. Moreover, using early exiting our model is capable of doing anytime prediction. This is especially useful for real-word energy-sensitive edge devices, where accuracy and complexity could be dynamically traded based on factors such as battery life, reliability, etc.Keywords
Funding Information
- National Science Foundation (2233893)
This publication has 50 references indexed in Scilit:
- Multi-attention fusion transformer for single-image super-resolutionScientific Reports, 2024
- PWLT: Pyramid Window-based Lightweight Transformer for image classificationComputers and Electrical Engineering, 2024
- No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling2023
- Adaptive Token Sampling for Efficient Vision Transformers2022
- Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning2022
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- A-ViT: Adaptive Tokens for Efficient Vision Transformer2022
- End-to-End Object Detection with Transformers2020
- Deep Residual Learning for Image Recognition2016
- ImageNet: A large-scale hierarchical image database2009