Abstract
Video-text retrieval becomes increasingly expensive when the number of videos grows, because standard cross-attention scales quadratically with sequence length. Existing methods usually improve either retrieval accuracy or computational efffciency, but rarely both at the same time. To address this issue, we present ViSA(Video-text interactive Synergistic Attention), a video-text interaction framework based on kernelized linear attention and subtraction-based fusion. The model reduces the cost of crossmodal interaction by pre-aggregating video features, so the overall computation grows linearly with the text and video lengths instead of forming a full attention matrix. At the same time, the differential design helps preserve useful discriminative information that is often weakened in linear attention. We evaluate ViSA on four benchmarks: MSR-VTT, LSMDC, DiDeMo, and VATEX. The results demonstrate that ViSA establishes a new state-of-the-art, outperforming strong CLIP-based baselines in accuracy while consistently using fewer GFLOPs. On MSR-VTT, it reaches 51.3% R@1 with 58 GFLOPs. Overall, ViSA provides an efffcient way to handle large-scale video-text retrieval without relying on quadratic softmax attention.