Spectral Reconstruction for Internet of Things Based on Parallel Fusion of CNN and Transformer

Authors:
Publisher website
Google Scholar
Abstract
Spectral imaging acquires more analyzable and distinguishable information than RGB imaging and has become an emerging technique powering Internet of Things (IoT). Thus, generating spectral images from existing RGB cameras is instrumental to high-level vision tasks. In this article, an IoT-oriented spectral reconstruction model with parallel fusion of convolutional neural network (CNN) and transformer (PFCT) is proposed to efficiently recover hyperspectral images (HSIs) from RGB counterparts, facilitating low-cost and nonhardware-specific spectral image acquisition. Recent works mainly utilize CNNs to extract local features by stacking more layers, ignoring the latent correlations of global features. In our PFCT network, we take advantage of lightweight CNNs to efficiently perceive local details, and exploit transformer blocks to fully capture the global context. Based on the architecture, a parallel fusion module is further designed to deeply interact and fuse the features obtained by CNN and Transformer in both directions. The final output spectral image is generated with same spatial sizes of the original image based on the deeply fused features. Consequently, the proposed PFCT network achieved high performance on four benchmark data sets compared to several state-of-the-art networks with a relatively small number of parameters.
Funding Information
  • National Natural Science Foundation of China (62076199)
  • Key Research and Development Project of Shaanxi Province (2022ZDLGY01-03, 2024GX-YBXM-129)
  • Key Scientific Research Program of Shaanxi Provincial Department of Education (23JY063)
  • Xi’an Science and Technology Research Plan (22GXFW0088)

This publication has 62 references indexed in Scilit: