Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding
Open Access
- 9 November 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 9754-9763
Abstract
Spatio-Temporal Video Grounding requires jointly localizing target objects across both temporal and spatial dimensions based on natural language queries, posing fundamental challenges for existing Multimodal Large Language Models (MLLMs). We identify two core challenges: entangled spatio-temporal alignment, arising from coupling two heterogeneous sub-tasks within the same autoregressive output space, and dual-domain visual token redundancy, where target objects exhibit simultaneous temporal and spatial sparsity, rendering the overwhelming majority of visual tokens irrelevant to the grounding query. To address these, we propose Bridge-STG, an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence. While decoupling is the natural solution to this entanglement, it risks creating a semantic gap between the temporal MLLM and the spatial decoder. Bridge-STG resolves this through two pivotal designs: the Spatio-Temporal Semantic Bridging (STSB) mechanism with Explicit Temporal Alignment (ETA) distills the MLLM's temporal reasoning context into enriched bridging queries as a robust semantic interface; and the Query-Guided Spatial Localization (QGSL) module leverages these queries to drive a purpose-built spatial decoder with multi-layer interactive queries and positive/negative frame sampling, jointly eliminating dual-domain visual token redundancy. Extensive experiments across multiple benchmarks demonstrate that Bridge-STG achieves state-of-the-art performance among MLLM-based methods. Bridge-STG improves average m\_vIoU from 26.4 to 34.3 on VidSTG and demonstrates strong cross-task transfer across various fine-grained video understanding tasks under a unified multi-task training regime.Keywords
Funding Information
- Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM103)
- Alibaba Innovation Research (AIR) Program (N\/A)
- Tencent WeChat Research Program (N\/A)
- SJTU-Huawei Explore X Gift Fund (N\/A)
This publication has 28 references indexed in Scilit:
- General Object Foundation Model for Images and Videos at Scale2024
- LISA: Reasoning Segmentation via Large Language Model2024
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding2024
- Collaborative Static and Dynamic Vision-Language Streams for Spatio-Temporal Video Grounding2023
- Query - Dependent Video Representation for Moment Retrieval and Highlight Detection2023
- ProTéGé: Untrimmed Pretraining for Video Temporal Grounding by Video Temporal Grounding2023
- TubeDETR: Spatio-Temporal Video Grounding with Transformers2022
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions2021
- Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences2020
- Generation and Comprehension of Unambiguous Object Descriptions2016