MIMFormer: Multiscale Inception Mixer Transformer for Hyperspectral and Multispectral Image Fusion
Open Access
- 22 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 17 (19391404) , 15122-15135
Abstract
The fusion of low-spatial-resolution hyperspectral image (LR-HSI) and high-spatial-resolution multispectral image (HR-MSI) provides an effective method to obtain high-spatial-resolution hyperspectral image (HR-HSI). However, existing hybrid fusion architectures combining Convolutional Neural Networks (CNNs) and Transformers face significant challenges. Sequential approaches struggle with simultaneous local and global modeling, while parallel approaches often result in information redundancy. In this paper, to meet diverse information demands at different layers, we propose a novel Multiscale Inception Mixer Transformer network (MIMFormer), a multiscale hybrid network based on the Inception structure integrating CNN and Transformer. The core of this network is the Multiscale Spatial Transformer (MST) structure, which enhances the detail richness of fused images by integrating local and global information at various scales. The Inception Spatial-spectral Mixer (ISSM) module within the MST leverages an Inception architecture and employs a spectral splitting mechanism to regulate spectral channel counts across different branches. This design allows the ISSM module to efficiently extract local spatial-spectral features through convolution and max pooling, while global features are captured using a self-attention mechanism, ensuring comprehensive feature fusion across spectral groups. Experimental results on three benchmark datasets and one real remote sensing dataset demonstrate that MIMFormer outperforms ten advanced fusion methods. The source code is publicly available at https://github.com/meiruni/MIMFormer .Keywords
Funding Information
- National Natural Science Foundation of China (42271487)
This publication has 53 references indexed in Scilit:
- HCT: a hybrid CNN and transformer network for hyperspectral image super-resolutionMultimedia Systems, 2024
- Spatio-spectral Cross-Attention Transformer for Hyperspectral image and Multispectral image fusionRemote Sensing Letters, 2023
- Reciprocal transformer for hyperspectral and multispectral image fusionInformation Fusion, 2023
- Learning a 3D-CNN and Transformer prior for hyperspectral image super-resolutionInformation Fusion, 2023
- Multiscale spatial–spectral transformer network for hyperspectral and multispectral image fusionInformation Fusion, 2023
- Multispectral and hyperspectral image fusion in remote sensing: A surveyInformation Fusion, 2022
- Hyperspectral and multispectral image fusion techniques for high resolution applications: a reviewEarth Science Informatics, 2021
- Recent advances and new guidelines on hyperspectral and multispectral image fusionInformation Fusion, 2021
- A new look at IHS-like image fusion methodsInformation Fusion, 2001
- Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniquesRemote Sensing of Environment, 1987