SimVidT: Video Transformer Acceleration via Exploiting Spatio-Temporal Similarity
- 1 November 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 45 (11) , 5960-5973
Abstract
Video transformers (VidTs) have demonstrated powerful spatio-temporal modelling capabilities in various video understanding tasks. However, their practical deployment remains constrained by high computational and memory overheads, primarily caused by the quadratic complexity of spatio-temporal self-attention and the intensive demands of processing high-frame-rate video streams. While recent efforts have attempted to leverage spatio-temporal similarity to eliminate redundant computation, they either rely on expensive similarity recognition or support only limited reuse, resulting in limited performance gains. In this work, we present a comprehensive analysis of the potential opportunities and challenges in exploiting spatiotemporal similarity within VidTs. Building upon these insights, we propose SimVidT, a novel VidT accelerator that leverages spatio-temporal similarity to reduce computational redundancy. SimVidT is supported by an algorithm and architecture codesign. At the algorithm level, we introduce a locality-aware similarity identification method and a redundancy-eliminated attention mechanism to efficiently identify and eliminate redundant computations at multiple granularities. At the architecture level, we develop a dedicated accelerator featuring a customized hierarchical memory management unit and specialized compute units, which adapts to the complex execution flow of VidTs and translates redundancy reduction into performance gains. We implement a cycle-accurate simulator for performance evaluation. Experimental results demonstrate that SimVidT achieves 1.7×speedup and 2.5× better energy efficiency on average compared to the state-of-the-art VidT accelerator CMC.Keywords
Funding Information
- National Key Research and Development Project from the Ministry of Science and Technology (2024YFA1211500)
This publication has 27 references indexed in Scilit:
- Hybrid multi-attention transformer for robust video object detectionEngineering Applications of Artificial Intelligence, 2024
- A video object detector with Spatio-Temporal Attention Module for micro UAV detectionNeurocomputing, 2024
- PublicVision: A Secure Smart Surveillance System for Crowd Behavior RecognitionIEEE Access, 2024
- ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention2023
- ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design2023
- Exploring Plain Vision Transformer Backbones for Object Detection2022
- A review of video surveillance systemsJournal of Visual Communication and Image Representation, 2021
- End-to-End Object Detection with Transformers2020
- PointConv: Deep Convolutional Networks on 3D Point Clouds2019
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015