Model-Informed Multistage Unsupervised Network for Hyperspectral Image Super-Resolution

Authors:
Top Cited Papers
Publisher website
Google Scholar
Abstract
By fusing a low-resolution hyperspectral image (LrMSI) with an auxiliary high-resolution multispectral image (HrMSI), hyperspectral image super-resolution (HISR) can generate a high-resolution hyperspectral image (HrHSI) economically. Despite the promising performance achieved by deep learning (DL), there are still two challenges remaining to be solved. First, most DL-based methods heavily rely on large-scale training triplets, which reduces them to limited generalization and poor practicability in real-world scenarios. Second, existing methods pursue higher performance by designing complex structures from off-the-shelf components while ignoring inherent information from the degradation model, hence leading to insufficient integration of domain knowledge and lower interpretability. To address those drawbacks, we propose a model-informed multistage unsupervised network, M2U-Net for short, by leveraging both deep image prior (DIP) and degradation model information. Generally, M2U-Net is built with a three-stage scheme, i.e., degradation information learning (DIL), initialized image establishment (IIE), and deep image generation (DIG) stages. The first stage is to exploit the deep information of the degradation model via a tiny network whose parameters and outputs will serve as guidance for the following two stages. Instead of feeding uninformed noise as input for stage three, the IIE stage aims to establish an initialized input with expressive HrHSI-relevant information by resorting to a spectral mapping learning network (SML-Net), thus facilitating the extraction of prior information and further magnifying the potential of DIP for high-quality reconstruction. Finally, we propose a dual U-shape network (Dual U-Net) as a powerful regularizer to capture image statistics, in which two U-Nets are coupled together by a cross-attention guidance (CAG) module to separately achieve spatial feature extraction and final image generation. The CAG module can incorporate abundant spatial information into the reconstruction process and, hence, guide the network toward a more plausible generation. Extensive experiments demonstrate the effectiveness of our proposed M2U-Net in terms of quantitative evaluation and visual quality. The code will be available at https://github.com/JiaxinLiCAS .
Funding Information
  • National Key Research and Development Program of China (2021YFB3900502)

This publication has 92 references indexed in Scilit: