Zhechen Yang

Waterbody extraction is of great significance for water resource investigation and monitoring. In addition to RGB bands, most common satellite images have a near-infrared (NIR) band. By combining these RGB-NIR bands, certain water, vegetation, and shadow indices can be calculated. The near-infrared band and these indices are very similar to the X modality in RGB+X data (common examples include RGB-D and RGB-Thermal). However, at present, no studies have thoroughly examined multimodal feature fusion from the RGB+X perspective in order to extract waterbodies with high precision. As a result, existing algorithms do not fully utilize satellite image information and have limited generalization ability. To overcome this limitation, we propose a dual-complexity backbone for waterbody extraction from the perspective of RGB+X data semantic segmentation. Its complex Transformer branch is used to extract RGB modality features, while its simple CNN branch is used to extract X modality features. This network structure can effectively capture multimodal, global, and local features in remote sensing images. It can also fully leverage the fact that the scale of RGB image datasets in computer vision is significantly larger than that of remote sensing waterbody extraction datasets. If a large pretrained model is used in the RGB branch, it is unnecessary to freeze the weights. Instead, both branches can be trained jointly, allowing the RGB branch to better adapt to the remote sensing waterbody extraction task without raising concerns that fine-tuning might undermine the pretrained model’s strong representation capability. We also propose two X modality configurations with strong generalization performance. To fully fuse multimodal features, we design a hybrid fusion module combining a CNN and a cross-attention mechanism. To integrate the multi-scale features, we employ a multi-scale Transformer structure in the RGB branch and design a multi-scale decoder. Our algorithm achieves state-of-the-art performance on the GID-5 dataset and competitive performance on the S1S2-Water dataset. Furthermore, it significantly outperforms existing methods in cross-dataset zero-shot transfer between the two datasets, with IoU/F1-score gains of 26.08%/27.33% on GID-5 and 38.74%/31.37% on S1S2-Water over previous SOTA methods. Our processing paradigm of modeling RGB-NIR remote sensing images as RGB+X data shows potential for generalization to other multi-modal remote sensing tasks. The dual-complexity backbone we design also has potential to be extended to other tasks that transfer large pretrained RGB models to remote sensing imagery with RGB-NIR four bands or even more spectral bands. We have open-sourced the code and trained models used in this research.

Plane instance segmentation from RGB-D data is a crucial research topic for many downstream tasks. However, most existing deep-learning-based methods utilize only information within the RGB bands, neglecting the important role of the depth band in plane instance segmentation. Based on EfficientSAM, a fast version of SAM, we propose a plane instance segmentation network called PlaneSAM, which can fully integrate the information of the RGB bands (spectral bands) and the D band (geometric band), thereby improving the effectiveness of plane instance segmentation in a multimodal manner. Specifically, we use a dual-complexity backbone, with primarily the simpler branch learning D-band features and primarily the more complex branch learning RGB-band features. Consequently, the backbone can effectively learn D-band feature representations even when D-band training data is limited in scale, retain the powerful RGB-band feature representations of EfficientSAM, and allow the original backbone branch to be fine-tuned for the current task. To enhance the adaptability of our PlaneSAM to the RGB-D domain, we pretrain our dual-complexity backbone using the segment anything task on large-scale RGB-D data through a self-supervised pretraining strategy based on imperfect pseudo-labels. To support the segmentation of large planes, we optimize the loss function combination ratio of EfficientSAM. In addition, Faster R-CNN is used as a plane detector, and its predicted bounding boxes are fed into our dual-complexity network as prompts, thereby enabling fully automatic plane instance segmentation. Experimental results show that the proposed PlaneSAM sets a new SOTA performance on the ScanNet dataset, and outperforms previous SOTA approaches in zero-shot transfer on the 2D-3D-S, Matterport3D, and ICL-NUIM RGB-D datasets, while only incurring a 10% increase in computational overhead compared to EfficientSAM.