State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding
Preprint
Abstract
Recently, pre-trained state space models have shown great potential for video classification, which sequentially compresses visual tokens in videos with linear complexity, thereby improving the processing efficiency of video data while maintaining high performance. To apply powerful pre-trained models to downstream tasks, prompt learning is proposed to achieve efficient downstream task adaptation with only a small number of fine-tuned parameters. However, the sequentially compressed visual prompt tokens fail to capture the spatial and temporal contextual information in the video, thus limiting the effective propagation of spatial information within a video frame and temporal information between frames in the state compression model and the extraction of discriminative information. To tackle the above issue, we proposed a State Space Prompting (SSP) method for video understanding, which combines intra-frame and inter-frame prompts to aggregate and propagate key spatiotemporal information in the video. Specifically, an Intra-Frame Gathering (IFG) module is designed to aggregate spatial key information within each frame. Besides, an Inter-Frame Spreading (IFS) module is designed to spread discriminative spatio-temporal information across different frames. By adaptively balancing and compressing key spatio-temporal information within and between frames, our SSP effectively propagates discriminative information in videos in a complementary manner. Extensive experiments on four video benchmark datasets verify that our SSP significantly outperforms existing SOTA methods by 2.76% on average while reducing the overhead of fine-tuning parameters.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Distribution-Aware Knowledge Aligning and Prototyping for Non-Exemplar Lifelong Person Re-IdentificationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding2025
- SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Prompting2025
- Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation2025
- Long Short-Term Knowledge Decomposition and Consolidation for Lifelong Person Re-IdentificationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Mitigate Catastrophic Remembering via Continual Knowledge Purification for Noisy Lifelong Person Re-Identification2024
- Compositional Prompting for Anti-Forgetting in Domain Incremental LearningInternational Journal of Computer Vision, 2024
- Distribution-Aware Knowledge Prototyping for Non-Exemplar Lifelong Person Re-Identification2024
- Exemplar-Free Lifelong Person Re-identification via Prompt-Guided Adaptive Knowledge ConsolidationInternational Journal of Computer Vision, 2024
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022