Abstract
Video recognition is a fundamental task in high-level video understanding. However, bridging semantic intent and visual perception remains challenging. Traditional label-supervised methods rely on annotations and therefore lack rich spatio-temporal reasoning. Multimodal Large Language Models (MLLMs) offer instruction-following flexibility. However, they often struggle to extract effective visual representations. Recent prompt-based approaches improve adaptability but still depend on static, task-agnostic prompts initialized from templates, GPT-like texts, or visual prototypes, which limits dynamic temporal reasoning. To address these limitations, we propose a spatio-temporal decoding framework via Instruction-Vision-Conditioned Prompts (IVCP), which leverages a large language model (LLM) to interpret task instructions and generate instruction-aware prompts conditioned on textual and visual contexts. Integrated into a transformer-based decoder, IVCP enables instruction-grounded spatio-temporal feature extraction and achieves superior accuracy across NTU-60/120, UCF-101, HMDB-51 and Something Something V2(SthSthV2) benchmarks.

This publication has 3 references indexed in Scilit: