An Energy-Efficient High Resolution Vision Transformer Processor Exploiting Token Similarity Beyond Token Merging
- 22 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
- Vol. 34 (1) , 118-129
Abstract
Token merging is a widely adopted technique in vision transformers (ViTs) to reduce computation with minimal accuracy degradation. However, in tasks that require fine-grained outputs, such as text-unified object recognition, aggressive merging reaches a limit due to accuracy loss. To address this, we introduce a method that exploits residual token similarity beyond the merging threshold using difference computing, which significantly improves energy efficiency. The proposed processor integrates the post merging residual difference (PMRD) algorithm, which first reduces 34% of tokens through merging, then applies difference computing to the remaining tokens. This results in a 31% reduction in effective bit operations by leveraging residual similarity. To further exploit the resulting bit-level sparsity, the processor employs sparsity-separated accumulation with sign-magnitude representation (SSA-SM) and a bi-directional dynamic memory allocation (BDA) to optimize both computation and memory access. These enhancements yield an additional 29.4% computational energy savings. Overall, the system achieves a $2.22\times $ improvement in energy efficiency and delivers 7.23 TOPS/W on the MS COCO dataset for text-unified object recognition.Keywords
Funding Information
- Institute of Information and Communications Technology Planning and Evaluation (IITP) Grant funded by Korean Government (MSIT) PIM Semiconductor Design Research Center (2022-0-01170)
This publication has 18 references indexed in Scilit:
- Cambricon-D: Full-Network Differential Acceleration for Diffusion Models2024
- ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers2024
- DynaPlasia: An eDRAM In-Memory Computing-Based Reconfigurable Spatial Accelerator With Triple-Mode CellIEEE Journal of Solid-State Circuits, 2023
- 29.3 An 8.09TOPS/W Neural Engine Leveraging Bit-Sparsified Sign-Magnitude Multiplications and Dual Adder Trees2023
- HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers2023
- ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design2023
- A 28nm 27.5TOPS/W Approximate-Computing-Based Transformer Processor with Asymptotic Sparsity Speculating and Out-of-Order Computing2022
- SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning2021
- LVIS: A Dataset for Large Vocabulary Instance Segmentation2019
- Zero-Shot Object Detection2018