TV-VAD: Temporal-frequency assisted video-text alignment for weakly supervised video anomaly detection

Authors:
  • HZHeng Zhang
    Heng Zhang
  • CYChenggang Yang
    Chenggang Yang
  • ZZZhengpeng Zhao
    Zhengpeng Zhao
  • YPYuanyuan Pu
    Yuanyuan Pu

Abstract
Weakly supervised video anomaly detection (WSVAD) aims to identify abnormal events in videos with only coarse video-level annotations. Previous methods primarily focus on modeling anomalies within the spatio-temporal domain of visual modality. However, how to capture global frame-level video features and fully utilize the anchoring role of textual features remains challenging. To address these dilemmas, a novel TV-VAD framework that employs Temporal-frequency assisted Video-text alignment for WSVAD is proposed. Specifically, the temporal-frequency video feature extraction module integrates a Transformer encoder with a dual-branch graph convolutional network to model temporal dynamics, while a Fourier frequency-domain feature recombination module is introduced to capture diverse expressions of anomalies. Subsequently, a video-text alignment block based on Kullback-Leibler divergence aligns the global frame-level video features with fused multi-modal representations, thereby better exploiting the anchoring role of the text modality. Furthermore, a multiclass loss function is employed to reformulate the traditional binary classification task of WSVAD into a multiclass classification task. Extensive experiments demonstrate that TV-VAD achieves state-of-the-art performance on two public benchmarks, achieving an AP of 85.61% on XD-Violence and an AUC of 88.12% on UCF-Crime. Code is available at: https://github.com/jinjinggu00/TV-VAD.

This publication has 24 references indexed in Scilit: