An End-to-End Framework Based on Vision-Language Fusion for Remote Sensing Cross-Modal Text-Image Retrieval
Open Access
- 13 May 2023
- journal article
- research article
- Published by MDPI AG in Mathematics
- Vol. 11 (10) , 2279
Abstract
Remote sensing cross-modal text-image retrieval (RSCTIR) has recently attracted extensive attention due to its advantages of fast extraction of remote sensing image information and flexible human–computer interaction. Traditional RSCTIR methods mainly focus on improving the performance of uni-modal feature extraction separately, and most rely on pre-trained object detectors to obtain better local feature representation, which not only lack multi-modal interaction information, but also cause the training gap between the pre-trained object detector and the retrieval task. In this paper, we propose an end-to-end RSCTIR framework based on vision-language fusion (EnVLF) consisting of two uni-modal (vision and language) encoders and a muti-modal encoder which can be optimized by multitask training. Specifically, to achieve an end-to-end training process, we introduce a vision transformer module for image local features instead of a pre-trained object detector. By semantic alignment of visual and text features, the vision transformer module achieves the same performance as pre-trained object detectors for image local features. In addition, the trained multi-modal encoder can improve the top-one and top-five ranking performances after retrieval processing. Experiments on common RSICD and RSITMD datasets demonstrate that our EnVLF can obtain state-of-the-art retrieval performance.Keywords
This publication has 32 references indexed in Scilit:
- Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking2019
- Stacked Cross Attention for Image-Text Matching2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- DOTA: A Large-Scale Dataset for Object Detection in Aerial Images2018
- Learning Semantic Concepts and Order for Image and Sentence Matching2018
- Learning Two-Branch Neural Networks for Image-Text Matching TasksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Exploring Models and Data for Remote Sensing Image Caption GenerationIEEE Transactions on Geoscience and Remote Sensing, 2017
- Remote Sensing Image Scene Classification: Benchmark and State of the ArtProceedings of the IEEE, 2017
- Deep semantic understanding of high resolution remote sensing image2016
- Orientation robust object detection in aerial images using deep convolutional neural network2015