UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 6307-6315
Abstract
Urbanization challenges underscore the necessity for effective satellite image-text retrieval methods to swiftly access specific information enriched with geographic semantics for urban applications. However, existing methods often overlook significant domain gaps across diverse urban landscapes, primarily focusing on enhancing retrieval performance within single domains. To tackle this issue, we present UrbanCross, a new framework for cross-domain satellite image-text retrieval. UrbanCross leverages a high-quality, cross-domain dataset enriched with extensive geo-tags from three countries to highlight domain diversity. It employs the Large Multimodal Model (LMM) for textual refinement and the Segment Anything Model (SAM) for visual augmentation, achieving fine-grained alignment of images, segments and texts, yielding a 10% improvement in retrieval performance. Additionally, UrbanCross incorporates an adaptive curriculum-based source sampler and a weighted adversarial cross-domain fine-tuning module, progressively enhancing adaptability across various domains. Extensive experiments confirm UrbanCross's superior efficiency in retrieval and adaptation to new urban environments, demonstrating an average performance increase of 15% over its version without domain adaptation mechanisms, effectively bridging the domain gap. Our code and dataset are publicly accessible at https://github.com/siruzhong/UrbanCross.Keywords
All Related Versions
Funding Information
- Guangzhou-HKUST (GZ) Joint Funding Program (2024A03J0620)
This publication has 16 references indexed in Scilit:
- Knowledge-Aided Momentum Contrastive Learning for Remote-Sensing Image Text RetrievalIEEE Transactions on Geoscience and Remote Sensing, 2023
- Segment Anything2023
- Multilanguage Transformer for Improved Text to Remote Sensing Image RetrievalIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022
- A Deep Semantic Alignment Network for the Cross-Modal Image-Text Retrieval in Remote SensingIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021
- Assessment of CNN-Based Methods for Poverty Estimation from Satellite Images2021
- Word–Sentence Framework for Remote Sensing Image CaptioningIEEE Transactions on Geoscience and Remote Sensing, 2020
- Smart City Development With Urban Transfer LearningComputer, 2018
- Exploring Models and Data for Remote Sensing Image Caption GenerationIEEE Transactions on Geoscience and Remote Sensing, 2017
- Combining satellite imagery and machine learning to predict povertyScience, 2016
- A Survey on Transfer LearningIEEE Transactions on Knowledge and Data Engineering, 2009