Mobile-friendly under-sampling single-pixel imaging based on a lightweight hybrid CNN-ViT architecture
Authors:
- Wenjie JiangWJWenjie Jiang
- Jinze SongJSJinze Song
- Zexi ChenZCZexi Chen
- 20 December 2024
- journal article
- research article
- Published by Optica Publishing Group in Optics Express
- Vol. 32 (27) , 48672-48682
Abstract
Single-pixel imaging (SPI) using deep learning networks, e.g., convolutional neural networks (CNNs) and vision transformers (ViTs), has made significant progress. However, these existing models, especially those based on ViT architectures, pose challenges due to their large number of parameters and computational loads, making them unsuitable for mobile SPI applications. To break through this limitation, we propose mobile ViT blocks to bring down the computation cost of traditional ViTs, and combine CNNs to design what we believe to be a novel lightweight CNN-ViT hybrid model for efficient and accurate SPI reconstruction. In addition, we also propose a general-purpose differential ternary modulation pattern scheme for deep learning SPI (DLSPI), which is training-friendly and hardware-friendly. Simulations and real experiments demonstrate that our method has higher imaging quality, lower memory consumption, and less computational burden than the state-of-the-art DLSPI methods.Keywords
Funding Information
- National Natural Science Foundation of China (62305144, 62341603, 62305179)
- Natural Science Foundation of Shandong Province (ZR2023QF093, ZR2023QF063, ZR2023MF103)
This publication has 32 references indexed in Scilit:
- Mid-infrared single-pixel imaging at the single-photon levelNature Communications, 2023
- Compressed ultrahigh-speed single-pixel imaging by swept aggregate patternsNature Communications, 2022
- Dual-color terahertz spatial light modulator for single-pixel imagingLight: Science & Applications, 2022
- Imaging biological tissue with high-throughput single-pixel compressive holographyNature Communications, 2021
- High-resolution sub-sampling incoherent x-ray imaging with a single-pixel detectorAPL Photonics, 2020
- Principles and prospects for single-pixel imagingNature Photonics, 2018
- Ghost Imaging Based on Deep LearningScientific Reports, 2018
- Deep learning for real-time single-pixel videoScientific Reports, 2018
- Deep-learning-based ghost imagingScientific Reports, 2017
- 3D Computational Imaging with Single-Pixel DetectorsScience, 2013