Abstract
Video transformers (VidTs) have demonstrated powerful spatio-temporal modelling capabilities in various video understanding tasks. However, their practical deployment remains constrained by high computational and memory overheads, primarily caused by the quadratic complexity of spatio-temporal self-attention and the intensive demands of processing high-frame-rate video streams. While recent efforts have attempted to leverage spatio-temporal similarity to eliminate redundant computation, they either rely on expensive similarity recognition or support only limited reuse, resulting in limited performance gains. In this work, we present a comprehensive analysis of the potential opportunities and challenges in exploiting spatiotemporal similarity within VidTs. Building upon these insights, we propose SimVidT, a novel VidT accelerator that leverages spatio-temporal similarity to reduce computational redundancy. SimVidT is supported by an algorithm and architecture codesign. At the algorithm level, we introduce a locality-aware similarity identification method and a redundancy-eliminated attention mechanism to efficiently identify and eliminate redundant computations at multiple granularities. At the architecture level, we develop a dedicated accelerator featuring a customized hierarchical memory management unit and specialized compute units, which adapts to the complex execution flow of VidTs and translates redundancy reduction into performance gains. We implement a cycle-accurate simulator for performance evaluation. Experimental results demonstrate that SimVidT achieves 1.7×speedup and 2.5× better energy efficiency on average compared to the state-of-the-art VidT accelerator CMC.
Funding Information
  • National Key Research and Development Project from the Ministry of Science and Technology (2024YFA1211500)

This publication has 27 references indexed in Scilit: