Maofan Zhao

Extracting 3D building information from monocular satellite sensor imagery remains a formidable challenge in the field of remote sensing. Multitask frameworks based on deep learning, which particularly for simultaneously predicting 2D building outlines and their respective heights using ortho-rectified satellite imagery, have shown promise in addressing this challenge. Height estimation is notably complex due to the absence of explicit height indicators, limited interaction between semantic-height features, and inadequate representation of building relationships. Moreover, the availability of data sources is a limiting factor for broader application. To overcome these issues, this study introduces an innovative instance-level multitask learning model (named BDH-Net) that leverages off-nadir perspectives and roof-to-footprint offset vectors to enhance modeling. This model comprises four key components: a pixel-wise feature extraction image encoder-decoder, a query transformer decoder, a multitask decoder, and a height decoder that employs intra-instance and inter-instance attention for precise building height estimation. Additionally, we pioneer the use of Google Earth imagery to construct an off-nadir satellite dataset with roof-to-footprint offset vectors specifically designed for building instance segmentation and height prediction, known as the BDH dataset. Comprehensive experiments demonstrate that the proposed BDH-Net significantly improves the accuracy of monocular 3D building data extraction by integrating roof-to-footprint offset vectors and leveraging context specific to each building instance. With the extensive coverage and regular updates of Google Earth imagery, BDH-Net holds substantial potential for wide-ranging and long-term applications. The source code of the proposed BDH-Net and the BDH dataset are publicly available at https://github.com/wishx98/BDHNet.

In the last decade, the rapid development of deep learning (DL) has made it possible to perform automatic, accurate, and robust change detection (CD) on large volumes of remote sensing images (RSIs). However, despite advances in CD methods, their practical application in real-world contexts remains limited because of diverse input data and the applicational context. For example, the collected RSIs can be time-series observations, and more informative results are required to indicate the time of change or the specific change category. Moreover, training a deep neural network (DNN) requires a massive amount of training samples, and in many cases these samples are difficult to collect. To address these challenges, various specific CD methods have been developed considering different application scenarios and training resources. Additionally, recent advancements in image generation, self-supervision, and visual foundation models (VFMs) have opened up new approaches to address the ”data-hungry” issue of DL-based CD. The development of these methods in broader application scenarios requires further investigation and discussion. Therefore, this article summarizes the literature methods for different CD tasks and the available strategies and techniques to train and deploy DL-based CD methods in sample-limited scenarios. We expect this survey to be able to provide new insights and inspiration for researchers in this field to develop more effective CD methods that can be applied in a wider range of contexts.

In the last decade, the rapid development of deep learning (DL) has made it possible to perform automatic, accurate, and robust Change Detection (CD) on large volumes of Remote Sensing Images (RSIs). However, despite advances in CD methods, their practical application in real-world contexts remains limited due to the diverse input data and the applicational context. For example, the collected RSIs can be time-series observations, and more informative results are required to indicate the time of change or the specific change category. Moreover, training a Deep Neural Network (DNN) requires a massive amount of training samples, whereas in many cases these samples are difficult to collect. To address these challenges, various specific CD methods have been developed considering different application scenarios and training resources. Additionally, recent advancements in image generation, self-supervision, and visual foundation models (VFMs) have opened up new approaches to address the 'data-hungry' issue of DL-based CD. The development of these methods in broader application scenarios requires further investigation and discussion. Therefore, this article summarizes the literature methods for different CD tasks and the available strategies and techniques to train and deploy DL-based CD methods in sample-limited scenarios. We expect that this survey can provide new insights and inspiration for researchers in this field to develop more effective CD methods that can be applied in a wider range of contexts.

Currently, the development of deep learning has had a positive impact on remote sensing image change detection (RSCD) tasks, but many current methods still face challenges in effectively processing global and local features, especially in the task of building change detection in high-resolution images containing complex scenes. The extraction of target-related features is typically difficult, and changes in scene conditions further increase the difficulty of identifying real changes. To address these challenges, we propose the interactive attention–enhanced change detection network (IAE-CDNet). We design the local–global interaction attention module (LGAM), which effectively establishes the interactive relationship between local and global features and realizes information interaction between branches, enhancing the ability to obtain architectural detail features. Additionally, our change perception attention enhancement module (CPAEM) enhances the feature perception ability of the real change area through the joint action of the internal comprehensive feature extractor (CFE) and the fusion attention mechanism (FAM). We conduct extensive experiments on three datasets. Results indicate that the evaluation indicators and performance of our IAE-CDNet are better than those of other state-of-the-art (SOTA) methods.

The mechanism of connecting multimodal signals through self-attention operation is a key factor in the success of multimodal Transformer networks in remote sensing data fusion tasks. However, traditional approaches assume access to all modalities during both training and inference, which can lead to severe degradation when dealing with modal-incomplete inputs in downstream applications. To address this limitation, we propose a novel approach to incomplete multimodal learning in the context of remote sensing data fusion and the multimodal Transformer. This approach can be used in both supervised and self-supervised pretraining paradigms. It leverages the additional learned fusion tokens in combination with modality attention and masked self-attention mechanisms to collect multimodal signals in a multimodal Transformer. The proposed approach employs reconstruction and contrastive loss to facilitate fusion in pretraining, while allowing for random modality combinations as inputs in network training. Experimental results show that the proposed method delivers state-of-the-art performance on two multimodal datasets for tasks, such as building instance/semantic segmentation and land-cover mapping when dealing with incomplete inputs during inference.

Change detection (CD) in high-resolution remote sensing has received large attention due to its wide range of applications. Many methods have been proposed in the literature and achieved excellent performance. However, they are often fully supervised, thus requiring abundant pixel-level labeled samples, which is time-consuming and labor-intensive. Especially compared to the common single-temporal interpretation, labeling bi-temporal images is often more complicated. Therefore, this study combines weakly supervised learning (WSL) to reduce label acquisition costs. But changed regions are small, fragmented, and similar to the background, which increase the gap between weakly supervised and fully supervised tasks. To address these difficulties, we explore self-supervised methods to construct a WSL framework based on image-level labels for general CD, termed WSLCD in this paper. First, we design a double-branch siamese network to derive embeddings and initial class attention maps (CAMs), which inputs the original image pair and the spatially transformed image pair. Second, mutual learning and equivariant regularization (MLER) is enforced on CAMs from different views, which implements consistency constraints in confusion regions and makes CAMs learn from each other based on saliency regions. Furthermore, prototype-based contrastive learning (PCL) is designed such that unreliable pixels can learn from prototypes computed from reliable pixels. PCL includes intra-view contrast and cross-view contrast depending on whether the prototypes and class embeddings are from the same view. With the above strategies, we narrow the gap between image-level weakly supervised CD and fully supervised CD. Experiments are conducted on three CD datasets, including CLCD, DSIFN and GCD. Our method achieves state-of-the-art performance on pseudo label generation and CD. The code is available at https://github.com/mfzhao1998/WSLCD.

With the development of high-resolution satellites, more and more attention has been paid to remote sensing (RS) scene classification. Convolutional neural networks (CNNs), which replace the traditional handcrafted features with a learning-based feature extraction mechanism, are widely used in scene classification. But CNNs are less effective in deriving long-range contextual relations, which limits the further improvement. Visual transformer (VT), an emerging image processing method, provides a new perspective for RS scene classification by directly acquiring long-range features. Although there have been limited works combining CNN and VT through simple concatenation, the collaborations between them are insufficient. To address these issues, we propose a local and long-range collaborative framework (L2RCF). First, we design a dual-stream structure to extract the local and long-range features. Second, a cross-feature calibration (CFC) module is designed for them to improve representation of the fusion features. Then, combining deep supervision (DS) and deep mutual learning (DML), a novel joint loss is proposed to enhance the dual-stream feature extractor and further improve the fused features. Finally, a two-stage semi-supervised training strategy is designed to improve performance with unlabeled samples. To demonstrate the effectiveness of L2RCF, we conducted experiments on three widely used RS scene classification data sets: RSSCN7, AID, and NWPU. The results show that L2RCF performs significantly better compared with some state-of-the-art scene classification methods.