Xinyang Pu

Existing open-world remote sensing foundation models rely on text-to-image alignment for open vocabulary semantic segmentation or object detection. However, semantic segmentation lacks instance-level delineation, while object detection fails to provide dense prediction for uncountable geographic features. Current approaches lack a unified solution for comprehensive panoptic understanding that jointly addresses semantic segmentation, instance segmentation, and object detection across all geographic elements. To bridge this gap, we propose PanoEarth, the first open-vocabulary panoptic segmentation framework for remote sensing images, which integrates these tasks within a unified architecture. Our framework incorporates a scalable training strategy to compensate missing background semantic information in instance segmentation data, enabling dense panoptic perception. Through the introduction of a domainspecific text embedding classifier to guide fine-tuning and establish precise mask query-to-category alignment, PanoEarth achieves competitive zero-shot open-vocabulary performance. We curate a large-scale panoptic dataset comprising 150k images spanning 111 geographic categories and millions of instances across landcover types, infrastructure elements, and fine-grained object subtypes. In zero-shot experiments, PanoEarth surpasses task-specific state-of-the-art models across multiple datasets and tasks. The joint co-training version enables unified extraction of 111 geographic categories, enabling hierarchical analysis from landcover mapping to fine-grained instance recognition. The source code is available at: https://github.com/xypu98/PanoEarth.

In the realm of artificial intelligence, the emergence of foundation models, backed by high computing capabilities and extensive data, has been revolutionary. Segment Anything Model (SAM), built on the Vision Transformer (ViT) model with millions of parameters and trained on its corresponding large-scale dataset SA-1B, excels in various segmentation scenarios relying on its significance of semantic information and generalization ability. Such achievement of visual foundation model stimulates continuous researches on specific downstream tasks in computer vision. The ClassWise-SAM-Adapter (CWSAM) is designed to adapt the high-performing SAM for landcover classification on space-borne Synthetic Aperture Radar (SAR) images. The proposed CWSAM freezes most of SAM's parameters and incorporates lightweight adapters for parameter efficient fine-tuning, and a classwise mask decoder is designed to achieve semantic segmentation task. This adapt-tuning method allows for efficient landcover classification of SAR images, balancing the accuracy with computational demand. In addition, the task specific input module injects low frequency information of SAR images by MLP-based layers to improve the model performance. Compared to conventional state-of-the-art semantic segmentation algorithms by extensive experiments, CWSAM showcases enhanced performance with fewer computing resources, highlighting the potential of leveraging foundational models like SAM for specific downstream tasks in the SAR domain. The source code is available at: https://github.com/xypu98/CWSAM.

Supervised fine-tuning methods (SFT) perform great efficiency on artificial intelligence interpretation in SAR images, leveraging the powerful representation knowledge from pre-training models. Due to the lack of domain-specific pre-trained backbones in SAR images, the traditional strategies are loading the foundation pre-train models of natural scenes such as ImageNet, whose characteristics of images are extremely different from SAR images. This may hinder the model performance on downstream tasks when adopting SFT on small-scale annotated SAR data. In this paper, an self-supervised learning (SSL) method of masked image modeling based on Masked Auto-Encoders (MAE) is proposed to learn feature representations of SAR images during the pre-training process and benefit the object detection task in SAR images of SFT. The evaluation experiments on the large-scale SAR object detection benchmark named SARDet-100k verify that the proposed method captures proper latent representations of SAR images and improves the model generalization in downstream tasks by converting the pre-trained domain from natural scenes to SAR images through SSL. The proposed method achieves an improvement of 1.3 mAP on the SARDet-100k benchmark compared to only the SFT strategies.

Deep learning models in satellite onboard enable real-time interpretation of remote sensing images, reducing the need for data transmission to the ground and conserving communication resources. As satellite numbers and observation frequencies increase, the demand for satellite onboard real-time image interpretation grows, highlighting the expanding importance and development of this technology. However, updating the extensive parameters of models deployed on the satellites for spaceborne object detection model is challenging due to the limitations of uplink bandwidth in wireless satellite communications. To address this issue, this paper proposes a method based on parameter-efficient fine-tuning technology with low-rank adaptation module. It involves training low-rank matrix parameters and integrating them with the original model’s weight matrix through multiplication and summation, thereby fine-tuning the model parameters to adapt to new data distributions with minimal weight updates. The proposed method combines parameter-efficient fine-tuning with full fine-tuning in the parameter update strategy of the oriented object detection algorithm architecture. This strategy enables model performance improvements close to full fine-tuning effects with minimal parameter updates. In addition, low rank approximation is conducted to explore intrinsic dimensions of parameter matrices in normal-size models. Extensive experiments conducted on the DOTAv1.0, HRSC2016, and DIOR-R datasets verify the effectiveness of the proposed method. By fine-tuning and updating only 12.4% of the model’s total parameters, it is able to achieve 97% to 100% of the performance of full fine-tuning models. Additionally, the reduced number of trainable parameters accelerates model training iterations and enhances the generalization and robustness of the oriented object detection model. The source code is available at: https://github.com/fudanxu/LoRA-Det.

The Segment Anything Model (SAM), a foundational model designed for promptable segmentation tasks, demonstrates exceptional generalization capabilities, making it highly promising for natural scene image segmentation. However, SAM's lack of pretraining on massive remote sensing images and its interactive structure limit its automatic mask prediction capabilities. In this paper, a Multi-Cognitive SAM-Based Instance Segmentation Model (MC-SAM SEG) is introduced to employ SAM on remote sensing domain. The SAM-Mona encoder utilizing the Multi-cognitive Visual Adapter (Mona) is conducted to facilitate SAM's transfer learning in remote sensing applications. The proposed method named MC-SAM SEG extracts high-quality features by fine-tuning the SAM-Mona encoder along with a feature aggregator. Subsequently, a pixel decoder and transformer decoder are designed for prompt-free mask generation and instance classification. The comprehensive experiments are conducted on the HRSID and WHU datasets for instance segmentation tasks on Synthetic Aperture Radar (SAR) images and optical remote sensing images respectively. The evaluation results indicate the proposed method surpasses other deep learning algorithms and verify its effectiveness and generalization.

Synthetic aperture radar (SAR) has emerged as a vital tool for ship monitoring due to its all-weather, all-day high-resolution imaging capabilities. In practical operations, the wide coverage and sparse ship distribution in very large full-scene SAR images pose challenges in terms of low efficiency and high false alarm rates. Traditional methods perform poorly in complex scenarios, while deep learning (DL) methods have high computation cost. This study proposes a fast progressive detection algorithm for ship targets in large SAR images, combining the advantages of traditional Non-DL methods and DL approaches. First, at a global scale, image preprocessing operations based on traditional methods are designed to quickly extract candidate regions. Then, at regional scale, an oriented ship detector is designed for refined ship detection within candidate regions. Finally, at individual-target scale, a false alarm discrimination network is constructed to further remove false alarms. Experimental results on GF-3 full-scene SAR images demonstrate that the proposed method can achieve minutes-level detection efficiency in images of billion-pixel-level size, while achieving high detection accuracy.

Amidst the landscape of contemporary remote sensing technology, the endeavor to detect and recognize aircraft within remote sensing images (RSIs) assumes pivotal strategic and practical significance. The complex nature of fine-grained aircraft recognition is a result of the intricate interplay between aircraft and their background environments, alongside category imbalance, which collectively lead to the emergence of a long-tail distribution within the dataset. However, experts proficient in RSIs interpretation can effectively address these challenges through the application of prior knowledge. This paper introduces the Aircraft Reasoning Network (ARNet), a framework tailored for aircraft detection and fine-grained recognition in RSIs, building upon prior knowledge employed in expert interpretation. Specifically, the Knowledge Reasoning Module (KRM) introduces a knowledge graph that incorporates both common and expert knowledge into the end-to-end network. Additionally, the network encompasses a Spatial Context Module (SCM) and an Airport Facility Relationship Module (AFRM). These components facilitate highly accurate detection and recognition of fine-grained aircraft in diverse environmental contexts by employing adaptive prior knowledge reasoning and optimizing target spatial location. Furthermore, an independent Aircraft Component Discrimination Module (ACDM) distinguishes aircraft based on their predominant component features, contributing to improved classification performance in both the few-shot and easily confused categories. Moreover, this paper introduces the AR-RSI dataset, a compilation of RSIs capturing fine-grained aircraft targets from diverse locations. The effectiveness and superiority of ARNet are exemplified on AR-RSI, achieving a minimum of 3.7 percentage higher mAP than the mainstream aircraft detection framework.

Synthetic aperture radar (SAR) offers all-weather and all-day high-resolution imaging, yet its unique imaging mechanism often necessitates expert interpretation, limiting its broader applicability. Addressing this challenge, this letter proposes a generative model that bridges SAR and optical imaging, facilitating the conversion of SAR images into more human-recognizable optical aerial images. This assists in the interpretation of SAR data, making it easier to recognize. Specifically, our model backbone is based on the recent diffusion models, which have powerful generative capabilities. We have innovatively tailored the diffusion model framework, incorporating SAR images as conditional constraints in the sampling process. This adaptation enables the effective translation from SAR to optical images. We conduct experiments on the satellite GF3 and SEN12 datasets and use structural similarity (SSIM) and Fréchet inception distance (FID) for quantitative evaluation. The results show that our model not only surpasses previous methods in quantitative evaluation but also significantly improves the visual quality of the generated images. This advancement underscores the model’s potential to enhance SAR image interpretation.

Three-dimensional (3D) reconstruction of complex targets on the ground from multi-view synthetic aperture radar (SAR) images is of great interests. The inherently-integrated forward-inverse architecture of the differentiable SAR renderer (DSR) provides a promising solution to the general inverse problem of SAR target reconstruction. In this context, the target’s shadow provides complementary information to its scattering image. Hence, this paper proposes a novel DSR-based target reconstruction approach using both the target image and its shadows. The capabilities of DSR are extended to generate not only target scattering images but also shadows. Furthermore, the gradients of the outputs, specifically illumination map and shadow map, with respect to the inputs, i.e., target geometry represented as a mesh, are derived. This enables us to develop a gradient-descent inverse approach for solving the general reconstruction problem. Extensive simulations and quantitative evaluations demonstrate that incorporating both the target scattering image and its shadows significantly improves the reconstruction performance. Moreover, our analyses indicate that achieving optimal reconstruction effects requires a minimum of 9 views with a relatively even distribution. Finally, the proposed algorithm is validated using real SAR images of vehicle targets.

Ship detection in low-quality Synthetic Aperture Radar (SAR) images poses a persistent challenge. Noise signals in complex environments disrupt imaging conditions, hindering SAR systems from acquiring precise target information, thereby significantly compromising the performance of detectors. Some methods mitigate interference via denoising techniques, while others introduce noise using classical methods to learn target features in the presence of noise. This conundrum is regarded as a cross-domain problem in this paper; a ship detection method in low-quality images is proposed to learn features of targets and shrink serious deterioration of detection performance by utilizing Generative Adversarial Networks (GANs). First, an image-to-image translation task is implemented using CycleGAN to generate low-quality SAR images with complex interference from the source domain to the target domain. Second, with the annotation inheritance, these generated SAR images participate in a training process to improve the detection accuracy and model robustness. Multiple experiments indicate that the proposed method conspicuously improves the detection performance and efficaciously reduces the missed detection rate in the SAR ship detection task. This cross-domain approach achieved outstanding improvements in the form of 11.0% mAP and 3.22% mAP on the GaoFen-3 ship dataset and SRSSD-V1.0, respectively. In addition, the characteristics and potentials of near-shore and off-shore SAR image reconstruction with style transfer based on Generative Adversarial Networks were explored and analyzed in this work.