NeuriCam: Key-Frame Video Super-Resolution and Colorization for IoT Cameras
Preprint
Abstract
We present NeuriCam, a novel deep learning-based system to achieve video capture from low-power dual-mode IoT camera systems. Our idea is to design a dual-mode camera system where the first mode is low-power (1.1 mW) but only outputs grey-scale, low resolution, and noisy video and the second mode consumes much higher power (100 mW) but outputs color and higher resolution images. To reduce total energy consumption, we heavily duty cycle the high power mode to output an image only once every second. The data for this camera system is then wirelessly sent to a nearby plugged-in gateway, where we run our real-time neural network decoder to reconstruct a higher-resolution color video. To achieve this, we introduce an attention feature filter mechanism that assigns different weights to different features, based on the correlation between the feature map and the contents of the input frame at each spatial location. We design a wireless hardware prototype using off-the-shelf cameras and address practical issues including packet loss and perspective mismatch. Our evaluations show that our dual-camera approach reduces energy consumption by 7x compared to existing systems. Further, our model achieves an average greyscale PSNR gain of 3.7 dB over prior single and dual-camera video super-resolution methods and 5.6 dB RGB gain over prior color propagation methods. Open-source code: https://github.com/vb000/NeuriCam.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Reference-based Video Super-Resolution Using Multi-Camera Video Triplets2022
- RRSGAN: Reference-Based Super-Resolution for Remote Sensing ImageIEEE Transactions on Geoscience and Remote Sensing, 2021
- MACSen: A Processing-In-Sensor Architecture Integrating MAC Operations Into Image Sensor for Ultra-Low-Power BNN-Based Intelligent Visual PerceptionIEEE Transactions on Circuits and Systems II: Express Briefs, 2020
- Image Super-Resolution by Neural Texture Transfer2019
- Battery-Free Wireless Video Streaming Camera System2019
- Video Enhancement with Task-Oriented FlowInternational Journal of Computer Vision, 2019
- Energy-Efficient Image Compressive Transmission for Wireless Camera NetworksIEEE Sensors Journal, 2016
- On Bayesian Adaptive Video Super ResolutionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Low-Power H.264 Video Compression Architectures for Mobile CommunicationIEEE Transactions on Circuits and Systems for Video Technology, 2009
- Example-based super-resolutionIEEE Computer Graphics and Applications, 2002