A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural Network
Open Access
Abstract
Light detection and ranging (LiDAR) technology, a cutting-edge advancement in mobile applications, presents a myriad of compelling use cases, including enhancing low-light photography, capturing and sharing 3D images of fascinating objects, and elevating the overall augmented reality (AR) experience. However, its widespread adoption has been hindered by the prohibitive costs and substantial power consumption associated with its implementation in mobile devices. To surmount these obstacles, this paper proposes a low-power, low-cost, single-photon avalanche detector (SPAD)-based system-on-chip (SoC) which packages the microlens arrays (MLAs) and a lightweight RGB-guided sparse depth imaging completion neural network for 3D LiDAR imaging. The proposed SoC integrates an 8 × 8 SPAD macropixel array with time-to-digital converters (TDCs) and a charge pump, fabricated using a 180 nm bipolar-CMOS-DMOS (BCD) process. Initially, the primary function of this SoC was limited to serving as a ranging sensor. A random MLA-based homogenizing diffuser efficiently transforms Gaussian beams into flat-topped beams with a 45° field of view (FOV), enabling flash projection at the transmitter. To further enhance resolution and broaden application possibilities, a lightweight neural network employing RGB-guided sparse depth complementation is proposed, enabling a substantial expansion of image resolution from 8 × 8 to quarter video graphics array level (QVGA; 256 × 256). Experimental results demonstrate the effectiveness and stability of the hardware encompassing the SoC and optical system, as well as the lightweight features and accuracy of the algorithmic neural network. The state-of-the-art SoC-neural network solution offers a promising and inspiring foundation for developing consumer-level 3D imaging applications on mobile devices.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network2018
- A 20-ch TDC/ADC Hybrid Architecture LiDAR SoC for 240 $\times$ 96 Pixel 200-m Range Imaging With Smart Accumulation Technique and Residue Quantizing SAR ADCIEEE Journal of Solid-State Circuits, 2018
- A 512 × 512 SPAD Image Sensor With Integrated Gating for Widefield FLIMIEEE Journal of Selected Topics in Quantum Electronics, 2018
- Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image2018
- Laser Spot Center Detection and Comparison TestPhotonic Sensors, 2018
- A 256×256 45/65nm 3D-stacked SPAD-based direct TOF image sensor for LiDAR applications with optical polar modulation for up to 18.6dB interference suppression2018
- Xception: Deep Learning with Depthwise Separable Convolutions2017
- Freeform microlens array homogenizer for excimer laser beam shapingOptics Express, 2016
- Deep Residual Learning for Image Recognition2016
- A 0.18-$\mu$m CMOS SoC for a 100-m-Range 10-Frame/s 200$\,\times\,$96-Pixel Time-of-Flight Depth SensorIEEE Journal of Solid-State Circuits, 2013