Exploring Self-Supervised Learning for Multi-Modal Remote Sensing Pre-Training via Asymmetric Attention Fusion
Open Access
- 10 December 2023
- journal article
- research article
- Published by MDPI AG in Remote Sensing
- Vol. 15 (24) , 5682
Abstract
Self-supervised learning (SSL) has significantly bridged the gap between supervised and unsupervised learning in computer vision tasks and shown impressive success in the field of remote sensing (RS). However, these methods have primarily focused on single-modal RS data, which may have limitations in capturing the diversity of information in complex scenes. In this paper, we propose the Asymmetric Attention Fusion (AAF) framework to explore the potential of multi-modal representation learning compared to two simpler fusion methods: early fusion and late fusion. Given that data from active sensors (e.g., digital surface models and light detection and ranging) is often noisier and less informative than optical images, the AAF is designed with an asymmetric attention mechanism within a two-stream encoder, applied at each encoder stage. Additionally, we introduce a Transfer Gate module to select more informative features from the fused representations, enhancing performance in downstream tasks. Our comparative analyses on the ISPRS Potsdam datasets, focusing on scene classification and segmentation tasks, demonstrate significant performance enhancements with AAF compared to baseline methods. The proposed approach achieves an improvement of over 7% in all metrics compared to randomly initialized methods for both tasks. Furthermore, when compared to early fusion and late fusion methods, AAF consistently outperforms in achieving superior improvements. These results underscore the effectiveness of AAF in leveraging the strengths of multi-modal RS data for SSL, opening doors for more sophisticated and nuanced RS analysis.Keywords
Funding Information
- National Natural Science Foundation of China (61971279, 62022054)
This publication has 44 references indexed in Scilit:
- Beyond RGB: Very high resolution urban remote sensing with multimodal deep networksISPRS Journal of Photogrammetry and Remote Sensing, 2018
- FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture2017
- Data Fusion and Remote Sensing: An ever-growing relationshipIEEE Geoscience and Remote Sensing Magazine, 2016
- Colorful Image Colorization2016
- Deep Residual Learning for Image Recognition2016
- Unsupervised Visual Representation Learning by Context Prediction2015
- Unsupervised Learning of Visual Representations Using Videos2015
- Fully convolutional networks for semantic segmentation2015
- SAR and optical remote sensing: Assessment of complementarity and interoperability in the context of a large-scale operational forest monitoring systemRemote Sensing of Environment, 2015
- Fusing high-resolution SAR and optical imagery for improved urban land cover study and classificationInternational Journal of Image and Data Fusion, 2010