Difference-Complementary Learning and Label Reassignment for Multimodal Semi-Supervised Semantic Segmentation of Remote Sensing Images
- 10 January 2025
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 566-580
Abstract
The feature fusion of optical and Synthetic Aperture Radar (SAR) images is widely used for semantic segmentation of multimodal remote sensing images. It leverages information from two different sensors to enhance the analytical capabilities of land cover. However, the imaging characteristics of optical and SAR data are vastly different, and noise interference makes the fusion of multimodal data information challenging. Furthermore, in practical remote sensing applications, there are typically only a limited number of labeled samples available, with most pixels needing to be labeled. Semi-supervised learning has the potential to improve model performance in scenarios with limited labeled data. However, in remote sensing applications, the quality of pseudo-labels is frequently compromised, particularly in challenging regions such as blurred edges and areas with class confusion. This degradation in label quality can have a detrimental effect on the model’s overall performance. In this paper, we introduce the Difference-complementary Learning and Label Reassignment (DLLR) network for multimodal semi-supervised semantic segmentation of remote sensing images. Our proposed DLLR framework leverages asymmetric masking to create information discrepancies between the optical and SAR modalities, and employs a difference-guided complementary learning strategy to enable mutual learning. Subsequently, we introduce a multi-level label reassignment strategy, treating the label assignment problem as an optimal transport optimization task to allocate pixels to classes with higher precision for unlabeled pixels, thereby enhancing the quality of pseudo-label annotations. Finally, we introduce a multimodal consistency cross pseudo-supervision strategy to improve pseudo-label utilization. We evaluate our method on two multimodal remote sensing datasets, namely, the WHU-OPT-SAR and EErDS-OPT-SAR datasets. Experimental results demonstrate that our proposed DLLR model outperforms other relevant deep networks in terms of accuracy in multimodal semantic segmentation.Keywords
Funding Information
- National Key Research and Development Program of China (2021YFB3900502)
This publication has 63 references indexed in Scilit:
- Delivering Arbitrary-Modal Semantic Segmentation2023
- ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation2022
- DMT: Dynamic mutual training for semi-supervised learningPattern Recognition, 2022
- MCANet: A joint semantic segmentation framework of optical and SAR images for land use classificationInternational Journal of Applied Earth Observation and Geoinformation, 2022
- OTA: Optimal Transport Assignment for Object Detection2021
- Semi-supervised Semantic Segmentation with Directional Context-aware Consistency2021
- Multi-scale deep feature learning network with bilateral filtering for SAR image classificationISPRS Journal of Photogrammetry and Remote Sensing, 2020
- Semi-Supervised Semantic Segmentation With Cross-Consistency Training2020
- Crop Classification based on Deep Learning in Northeast China using SAR and Optical Imagery2019
- Fully convolutional networks for semantic segmentation2015