OASIS: Optimized Lightweight Autoencoder System for Distributed In-Sensor computing
Preprint
Abstract
In-sensor computing, which integrates computation directly within the sensor, has emerged as a promising paradigm for machine vision applications such as AR/VR and smart home systems. By processing data on-chip before transmission, it alleviates the bandwidth bottleneck caused by high-resolution, high-frame-rate image transmission, particularly in video applications. We envision a system architecture that integrates a CMOS image sensor (CIS) with a logic chip via advanced packaging, where the logic chip processes early-stage deep neural network (DNN) layers. However, its limited compute and memory make deploying advanced DNNs challenging. A simple solution is to split the model, executing the first part on the logic chip and the rest off-chip. However, modern DNNs require multiple layers before dimensionality reduction, limiting their ability to achieve the primary goal of in-sensor computing: minimizing data bandwidth. To address this, we propose a dual-branch autoencoder-based vision architecture that deploys a lightweight encoder on the logic chip while the task-specific network runs off-chip. The encoder is trained using a triple loss function: (1) task-specific loss to optimize accuracy, (2) entropy loss to enforce compact and compressible representations, and (3) reconstruction loss (mean-square error) to preserve essential visual information. This design enables a four-order-of-magnitude reduction in output activation dimensionality compared to input images, resulting in a $2{-}4.5\times$ decrease in energy consumption, as validated by our hardware-backed semi-analytical energy models. We evaluate our approach on CNN and ViT-based models across applications in smart home and augmented reality domains, achieving state-of-the-art accuracy with energy efficiency of up to 22.7 TOPS/W.
Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- PiPSim: A Behavior-Level Modeling Tool for CNN Processing-in-Pixel AcceleratorsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023
- A Reconfigurable Convolution-in-Pixel CMOS Image Sensor ArchitectureIEEE Transactions on Circuits and Systems for Video Technology, 2022
- Near-Sensor Distributed DNN Processing for Augmented and Virtual RealityIEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2021
- How computer vision can facilitate flood management: A systematic reviewInternational Journal of Disaster Risk Reduction, 2021
- A 7-nm Compute-in-Memory SRAM Macro Supporting Multi-Bit Input, Weight and Output and Achieving 351 TOPS/W and 372.4 GOPSIEEE Journal of Solid-State Circuits, 2020
- Ultrafast machine vision with 2D material neural network image sensorsNature, 2020
- 2.3 A 220GOPS 96-Core Processor with 6 Chiplets 3D-Stacked on an Active Interposer Offering 0.6ns/mm Latency, 3Tb/s/mm2 Inter-Chiplet Interconnects and 156mW/mm2@ 82%-Peak-Efficiency DC-DC Converters2020
- Processing Near Sensor Architecture in Mixed-Signal Domain With CMOS Image Sensor of Convolutional-Kernel-Readout MethodIEEE Transactions on Circuits and Systems I: Regular Papers, 2019
- CMOS image sensors: electronic camera-on-a-chipIEEE Transactions on Electron Devices, 1997
- A Method for the Construction of Minimum-Redundancy CodesProceedings of the IRE, 1952