Multiscale Temporal Modeling with Semantic Enhancement for Zero-shot Video Action Recognition

Authors:

Abstract
Zero-shot video action recognition relies on effective alignment between visual and semantic features to recognize actions from unseen categories. However, existing CLIP-based video methods typically rely on frame-level representations with limited cross-frame interaction, making it difficult to maintain cross-frame consistency and leading to wrong predictions. To address this, we propose MTM-CLIP, a framework for multiscale temporal modeling. It explicitly captures temporal action patterns across short-term, mid-term, and global-term windows in an adaptive manner. In addition, we introduce a conditional semantic enhancement mechanism, where learnable gated prompts are integrated into the CLIP visual encoder to modulate cross-frame temporal features. These features are refined via semantic guidance to enhance discriminative features at key temporal positions. By jointly modeling cross-frame temporal interactions and conditional semantic modulation, MTM-CLIP improves vision-language alignment and zero-shot generalization. Extensive experiments show that MTM-CLIP significantly outperforms prior CLIP-based methods across multiple zero-shot video action recognition benchmarks.