Mobile-friendly under-sampling single-pixel imaging based on a lightweight hybrid CNN-ViT architecture

Authors:
  • WJWenjie Jiang
    Wenjie Jiang
  • JSJinze Song
    Jinze Song
  • ZCZexi Chen
    Zexi Chen

Abstract
Single-pixel imaging (SPI) using deep learning networks, e.g., convolutional neural networks (CNNs) and vision transformers (ViTs), has made significant progress. However, these existing models, especially those based on ViT architectures, pose challenges due to their large number of parameters and computational loads, making them unsuitable for mobile SPI applications. To break through this limitation, we propose mobile ViT blocks to bring down the computation cost of traditional ViTs, and combine CNNs to design what we believe to be a novel lightweight CNN-ViT hybrid model for efficient and accurate SPI reconstruction. In addition, we also propose a general-purpose differential ternary modulation pattern scheme for deep learning SPI (DLSPI), which is training-friendly and hardware-friendly. Simulations and real experiments demonstrate that our method has higher imaging quality, lower memory consumption, and less computational burden than the state-of-the-art DLSPI methods.
Funding Information
  • National Natural Science Foundation of China (62305144, 62341603, 62305179)
  • Natural Science Foundation of Shandong Province (ZR2023QF093, ZR2023QF063, ZR2023MF103)

This publication has 32 references indexed in Scilit: