MsIFT: Multi-Source Image Fusion Transformer
Open Access
- 19 August 2022
- journal article
- research article
- Published by MDPI AG in Remote Sensing
- Vol. 14 (16) , 4062
Abstract
Multi-source image fusion is very important for improving image representation ability since its essence relies on the complementarity between multi-source information. However, feature-level image fusion methods based on the convolution neural network are impacted by the spatial misalignment between image pairs, which leads to the semantic bias in merging features and destroys the representation ability of the region-of-interests. In this paper, a novel multi-source image fusion transformer (MsIFT) is proposed. Due to the inherent global attention mechanism of the transformer, the MsIFT has non-local fusion receptive fields, and it is more robust to spatial misalignment. Furthermore, multiple classification-based downstream tasks (e.g., pixel-wise classification, image-wise classification and semantic segmentation) are unified in the proposed MsIFT framework, and the fusion module architecture is shared by different tasks. The MsIFT achieved state-of-the-art performances on the image-wise classification dataset VAIS, semantic segmentation dataset SpaceNet 6 and pixel-wise classification dataset GRSS-DFC-2013. The code and trained model are being released upon the publication of the work.Keywords
Funding Information
- National Natural Science Foundation of China (62071466, 6142A010402, 2018GXNSFBA281086)
- Fund of National Key Laboratory of Science and Technology on Remote Sensing Information and Imagery Analysis, Beijing Research Institute of Uranium Geology (62071466, 6142A010402, 2018GXNSFBA281086)
- Guangxi Natural Science Foundation (62071466, 6142A010402, 2018GXNSFBA281086)
This publication has 35 references indexed in Scilit:
- Multimodal Deep Learning for Robust Recognizing Maritime Imagery in the Visible and Infrared Spectrums2018
- Multisource Remote Sensing Data Classification Based on Convolutional Neural NetworkIEEE Transactions on Geoscience and Remote Sensing, 2017
- Multiple features learning for ship classification in optical imageryMultimedia Tools and Applications, 2017
- Pyramid Scene Parsing Network2017
- Building extraction from multi-source remote sensing images via deep deconvolution neural networks2016
- Deep Residual Learning for Image Recognition2016
- Fusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature LearningIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015
- Local Binary Patterns and Extreme Learning Machine for Hyperspectral Imagery ClassificationIEEE Transactions on Geoscience and Remote Sensing, 2015
- Report on the 2013 IEEE GRSS Data Fusion Contest: Fusion of Hyperspectral and LiDAR Data [Technical Committees]IEEE Geoscience and Remote Sensing Magazine, 2013
- ImageNet: A large-scale hierarchical image database2009