Xiu Li

In recent years, cross-modal text-image retrieval in remote sensing has gained prominence as a research focus due to its potential to provide abundant, inclusive, and multi-perspective information. However, existing methods usually focus on salient features, but these salient features cannot describe the image or text completely, resulting in the loss of some important details and discriminable information. In this article, a Whole Semantic Sparse Coding Network (WSSCN) is proposed for remote sensing image-text retrieval to build a complete and reliable features description for further improving the performance of the retrieval model. Specifically, the WSSCN first designs a Whole Semantic Sparse Representation Coding (WSSRC) module by constructing a robust semantic library to transform the dense features matrix of the image and text into a whole semantic sparse matrix that enables multiple semantic decoupling and leads to a more precise and detailed features expression. Afterwards, the Intra- and Inter-Modal Consistency (IIMC) module is devised to improve the intra-modal and inter-modal consistency of the whole semantic sparse representation from different models. Finally, the Salient and Whole Semantic Adaptive Learning (SWSAL) loss is proposed to focus on salient information or whole semantic information by calculating an adjusted parameter. Quantitative and qualitative experiments are performed on four extensive datasets for cross-modal retrieval in remote sensing to showcase the notable effectiveness achieved by implementing a whole semantic sparse coding network.

Sea surface temperature (SST) is a crucial metric in marine science, playing a pivotal role in forecasting and analyzing changes in the marine environment. However, remote sensing technologies often encounter issues where SST images are obscured by clouds, leading to data loss, thereby impacting marine environment prediction efficacy. Although many deep learning methods currently exist for reconstructing SST images, most focus on handling this task within the image domain, making it challenging to adapt to the chaotic nature of ocean systems. Additionally, most methods only model at a single scale, which limits their ability to effectively capture the complex multi-scale features in SST data. Therefore, this study proposes MSF_FNO, an image completion method based on Multi-Scale Fourier Fusion Neural Operator. MSF_FNO integrates multi-scale feature fusion and frequency domain neural operator technology to effectively overcome the limitations of single-scale feature processing and image domain reconstruction in existing methods. This approach not only captures SST frequency domain information and extracts structured features of SST images but also extracts critical features across multiple scales, ensuring global consistency and detailed features in reconstruction results. Experiments on the National Satellite Ocean Application Service (NSOAS) datasets demonstrate that MSF_FNO outperforms state-of-the-art (SOTA) methods in terms of reconstruction quality and robustness.

Recently, remote sensing text-image retrieval has gained increasing attention among researchers due to its capability to provide abundant, inclusive, and multi-perspective information. To extract the salient representation of the two modalities and realize information alignment, existing methods usually apply convolution calculation, matrix multiplication, and attention mechanisms to model features in the spatial domain. However, unlike natural images, remote sensing data contain a considerable amount of noise, and the spatial domain calculation methods will lead to information smoothing and noise exchange, ultimately reducing the robustness of feature representation. In this article, a Frequency and Spatial-domain Saliency Network (FSSN) is proposed by extracting saliency features in both frequency and spatial domains to further enhance the efficiency of retrieval networks. Specifically, the FSSN first designs a Frequency-domain Intra-modal Low-pass Filter (FILF) by using the Fourier Transform to convert the spatial domain representation into the frequency domain representation, and leverage low-pass filtering to filter out the noise information of the image. Afterwards, the Space-domain Inter-modal Boundary-based Saliency module (SIBS) is devised, which fully utilizes the positive and negative boundaries and draws an innovative assessment mechanism to automatically recognize the effective regions and words without noise shifting in the spatial domain. Finally, the Frequency and Spatial-domain Saliency Fusion (FSSF) is designed to realize the effective integration of features obtained from the frequency and spatial domain according to the consistency of the identity masks. Quantitative and qualitative experiments are implemented on four remote sensing benchmarks to showcase the notable effectiveness achieved by the joint modeling network of multiple domains. Specifically, the proposed FSSN outperforms the best model RemoteClip by 7.58% of the mR evaluation matrix on the UCM dataset.

Cross-modality based on remote sensing (RS) text-image retrieval has gained increasing attention in recent years due to its ability to leverage the rich semantics of images and the understandability of text to provide a more comprehensive description. Existing cross-modal retrieval methods typically apply self-attention or cross-attention mechanisms to identify important information in RS data, but they ignore the multi-view perception characteristic of geographical space in RS images. As a result, these retrieval models fail to locate the correct perspective in images according to the query text, ultimately leading to incorrect matching. In this work, a Cross-modal Progressive Perspective Matching Network (CPPMN) is proposed for remote sensing image-text retrieval by establishing a progressive perspective matching mechanism and semantic alignment to further improve the performance of the retrieval model. Specifically, the CPPMN framework consists of three core modules: the Compensation Network for Full Perspective Modeling (CN_FPM), the Graph Transformation for Individual Perspective Modeling (GT_IPM), and the Cascaded Transformer for Cross-modal Semantic Alignment (CT_CSA). The CN_FPM module utilizes all positive text samples as supervision signals to guide the feature extraction training process, aiming to capture full perspective information from images. Subsequently, the GT_IPM module transforms implicit-perspective feature representations into explicit-perspective cross-modal relationship graphs. This transformation enables the identification of specific perspective locations within the image according to the query sentence by analyzing graph density and connectivity. Finally, the CT_CSA module comprises a cascaded Transformer network that aligns features at the semantic level between cross-modal data The quantitative and qualitative experiments are conducted on four large-scale remote sensing cross-modal retrieval datasets to demonstrate the significant performance of adopting the progressive perspective matching mechanism and semantic alignment strategy.

modal retrieval is vital at the intersection of vision and language. Specifically, remote sensing image-text retrieval enhances our understanding of complex remote sensing content by combining multi-perspective visual information with concise textual descriptions and has increasingly become a hotspot for research. Existing prompts typically emphasize either global or local information, which fails to excavate or fully leverage the effective information of cross-modal data, resulting in the subpar performance of retrieval models. To address these limitations, we propose a novel method called Strong and Weak Prompt Engineering (SWPE) for remote sensing image-text retrieval. Specifically, SWPE employs the Strong and Weak Prompt Generation (SWPG) module to generate fine-grained and global category semantic prompts via an attention mechanism and a pretrained classification model. The Prompt-guided Feature Finetuning (PFF) module then refines the prompt information using a Transformer architecture, integrating the refined prompts with high-level image and text features to enhance both fine-grained details and global semantics. Finally, the Adaptive Hard Sample Elimination (AHSE) module optimizes the triplet loss function by training the model with negative sample pairs of varying difficulty, assigning higher weights to simpler pairs. Extensive quantitative and qualitative experiments on four remote sensing benchmarks validate the superior effectiveness of SWPE.

Remote sensing cross-modal image-text retrieval has attracted increasing attention due to its important roles in multiple domains. Existing methods perform salient modeling for the feature that has high relevance between different modalities. However, most works consider the relevance between different modalities but ignore the irrelevance between different modalities, resulting in incomplete modeling of the relevance and irrelevance between different modalities. In this paper, we propose a Relevance and Irrelevance Considered Subspace Mapping Neural Networks (RIR-SMNNs) to simultaneously consider the relevance and irrelevance between different modalities. Specifically, we first utilize Multiscale Image Feature Extraction (MIFE) and Multiscale Text Feature Extraction (MTFE) to extract the multiscale feature of image and text. Then, we perform Local Space Building Module (LSB), which constructs local space that realizes scale alignment. Finally, we perform the Relevance and Irrelevance Local Space Mapping (RIRLSM) to consider the relevance and irrelevance of different modalities in multiple spaces. Experimental results on several remote sensing datasets demonstrate our model outperforms the state-of-the-art approaches.