Yusheng Lian

Compared with traditional hyperspectral imaging, computational spectral imaging (CSI) has the advantage of snapshot imaging with high spatial and temporal resolution, which has attracted considerable attention. The core challenge of CSI is to achieve computational imaging reconstruction from a single 2D measurement image to the corresponding 3D spatial–hyperspectral image (HSI). Existing reconstruction methods still face problems in exploring spatial–spectral cross correlation, leading to significant spatial–spectral distortion. Furthermore, due to neglect of multi-scale feature reconstruction, their reconstruction quality still needs to be improved. In this paper, to solve the above problems, we propose a spatial–spectral cross-attention-driven network (SSCA-DN). In SSCA, a proposed multi-scale feature aggregation (MFA) module and a spectral-wise transformer (SpeT) are used for multi-scale spatial feature reconstruction and long-range spectral feature reconstruction, respectively. Using spatial attention and spectral attention to interactively guide the reconstruction of the target HSI in spectral and spatial dimensions, the proposed SSCA models spatial–spectral cross correlation with considering multi-scale features. Using the SSCA as a basic module, a novel SSCA-DN network is constructed, in which a proposed supervised preliminary reconstruction subnetwork (SPRNet) learns the generalized prior, and a proposed unsupervised multi-scale feature fusion and refinement subnetwork (UMFFRNet) learns the specific prior. The SSCA module ensures that the learned generalized and specific priors can capture the spatial–spectral cross correlation while considering multi-scale features. In addition, in UMFFRNet, driven by MFA and SSCA, a novel multi-scale fusion and refinement mechanism for multi-level adjacent features is proposed to effectively model the correlation between adjacent level features and the multi-scale spatial–spectral cross correlation, which further improves the reconstruction accuracy. Extensive experiments show that our method achieves state-of-the-art performance on both simulated and real datasets.

Unsupervised spectral reconstruction (SR) aims to recover the hyperspectral image (HSI) from corresponding RGB images without annotations. Existing SR methods achieve it from a single RGB image, hindered by the significant spectral distortion. Although several deep learning-based methods increase the SR accuracy by adding RGB images, their networks are always designed for other image recovery tasks, leaving huge room for improvement. To overcome this problem, we propose a novel, to our knowledge, approach that reconstructs the HSI from a pair of RGB images captured under two illuminations, significantly improving reconstruction accuracy. Specifically, an SR iterative model based on two illuminations is constructed at first. By unfolding the proximal gradient algorithm solving this SR model, an interpretable unsupervised deep network is proposed. All the modules in the proposed network have precise physical meanings, which enable our network to have superior performance and good generalization capability. Experimental results on two public datasets and our real-world images show the proposed method significantly improves both visually and quantitatively as compared with state-of-the-art methods.

Fusing a low spatial resolution hyperspectral image with a high spatial resolution multispectral image has become popular for generating a high spatial resolution hyperspectral image (HR-HSI). Most methods assume that the degradation information from high resolution to low resolution is known in spatial and spectral domains. Conversely, this information is often limited or unavailable in practice, restricting their performance. Furthermore, existing fusion methods still face the problem of insufficient exploration of the cross-interaction between the spatial and spectral domains in the HR-HSI, leaving scope for further improvement. This paper proposes an unsupervised Hybrid Network of Transformer and CNN (uHNTC) for blind HSI-MSI fusion. The uHNTC comprises three subnetworks: a transformer-based feature fusion subnetwork (FeafusFomer) and two CNN-based degradation subnetworks (SpaDNet and SpeDNet). Considering the strong multi-level spatio-spectral correlation between the desired HR-HSI and the observed images, we design a Multi-level Cross-feature Attention (MCA) mechanism in FeafusFormer. By incorporating the hierarchical spatio-spectral feature fusion into the attention mechanism in the transformer, the MCA globally keeps a high spatio-spectral cross-similarity between the recovered HR-HSI and observed images, thereby ensuring the high cross-interaction of the recovered HR-HSI. Subsequently, the characteristics of degradation information are utilized to guide the design of the SpaDNet and SpeDNet, which helps FeafusFormer accurately recover the desired HR-HSI in complex real-world environments. Through an unsupervised joint training of the three subnetworks, uHNTC recovers the desired HR-HSI without pre-known degradation information. Experimental results on three public datasets and a WorldView-2 images show that the uHNTC outperforms ten state-of-the-art fusion methods. Code available: https://github.com/Caoxuheng/HIFtool.