IVCP: Instruction-Guided Video Context Prompting for Video Action Recognition
Authors:
- Shichao KanSKShichao Kan
- Wentao LiWLWentao Li
- Peifu WangPWPeifu Wang
- Yi-Gang CenYCYi-Gang Cen
- Linna ZhangLZLinna Zhang
- Lihui CenLCLihui Cen
Preprint
- 14 April 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Video recognition is a fundamental task in high-level video understanding. However, bridging semantic intent and visual perception remains challenging. Traditional label-supervised methods rely on annotations and therefore lack rich spatio-temporal reasoning. Multimodal Large Language Models (MLLMs) offer instruction-following flexibility. However, they often struggle to extract effective visual representations. Recent prompt-based approaches improve adaptability but still depend on static, task-agnostic prompts initialized from templates, GPT-like texts, or visual prototypes, which limits dynamic temporal reasoning. To address these limitations, we propose a spatio-temporal decoding framework via Instruction-Vision-Conditioned Prompts (IVCP), which leverages a large language model (LLM) to interpret task instructions and generate instruction-aware prompts conditioned on textual and visual contexts. Integrated into a transformer-based decoder, IVCP enables instruction-grounded spatio-temporal feature extraction and achieves superior accuracy across NTU-60/120, UCF-101, HMDB-51 and Something Something V2(SthSthV2) benchmarks.Keywords
This publication has 3 references indexed in Scilit:
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity UnderstandingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Long Short-Term MemoryNeural Computation, 1997