VisproVG: Confidence-Aware Visual Prompting for Remote Sensing Visual Grounding

Authors:
Publisher website
Google Scholar
Abstract
Remote sensing visual grounding (RSVG) models commonly rely on backbones pretrained on natural image datasets. However, the inherent domain gap between natural images and remote sensing imagery significantly degrades model transferability, particularly when RSVG training samples are limited. Existing approaches primarily focus on enhancing localization accuracy through multi-scale visual–textual alignment and process each image–text pair independently, which limits their generalization capability under constrained supervision. In this work, we propose VisproVG, a novel framework that explicitly incorporates category-aware visual priors into the linguistic stream to mitigate domain shift and data scarcity. Specifically, VisproVG introduces a sample-level visual prompting mechanism that retrieves instance-level visual cues from a dynamically maintained, category-specific memory bank, which serve as stable and interpretable visual priors. To further improve semantic consistency across feature scales, we incorporate a lightweight cross-scale semantic refinement module that performs hierarchical top–down refinement. Beyond architectural design, we conduct a systematic analysis of the relationship between decoder confidence and localization accuracy. The analysis indicates that low-confidence predictions, while accounting for only a small fraction of test samples, exhibit substantially greater potential for improvement. Motivated by this observation, we introduce a confidence-guided adaptive test-time augmentation (TTA) strategy that selectively applies augmentation to uncertain samples, achieving a favorable balance between accuracy gains and computational efficiency. Extensive experiments on three RSVG benchmarks demonstrate the effectiveness of the proposed VisproVG. Notably, on the small-scale RSVG-HR dataset, VisproVG surpasses the second-best method by 5.05% meanIoU, highlighting its robustness under limited training data. Our code are available at https://github.com/XiaowZhang00/VisproVG.
Funding Information
  • National Natural Science Foundation of China (U22B2054)
  • National Natural Science Foundation of China (62076192)
  • National Natural Science Foundation of China (62276199)
  • National Natural Science Foundation of China (62431020)
  • National Natural Science Foundation of China (62276201)
  • Higher Education Discipline Innovation Project
  • Program for Cheung Kong Scholars and Innovative Research Team in University (IRT 15R53)
  • Science and Technology Innovation Project from the Chinese Ministry of Education
  • National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi'an Jiaotong University (HMHAI-202404)
  • National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi'an Jiaotong University (HMHAI-202405)

This publication has 39 references indexed in Scilit: