High Dynamic Range Imaging for Dynamic Scenes Based on Multi-Level Spike Camera
- 9 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (6) , 5394-5406
Abstract
Spike camera is a retina-inspired neuromorphic camera which can capture dynamic scenes of high-speed motion by firing a continuous stream of spikes at an extremely high temporal resolution. The limitation in the current design is that each spike only represents the arrival of a fixed amount of photons. It can not deal with strong light areas in which the amount of accumulated photons reaches the pre-specified threshold multiple times within a single readout interval. In this paper, we propose a new spike camera model of high-speed imaging for high dynamic range scenarios. In this scheme, each pixel accumulates the incoming photons persistently and generates a new type of spike stream in which each spike symbol may be associated with different levels, indicating the arrival of different amounts of photons since the last readout. This enables the camera to support dynamic scenes with wider dynamic range. To achieve this, we propose a two-level buffer mechanism, one for photon accumulation and one for spike-firing encoding. We use a register to hold the number of spike-firings which has not been read out yet. At each readout time, the major part in the counter is read out via a carefully designed exponential encoding and the counter is updated. Such encoding and readout strategy enables a very efficient expansion of the dynamic range using a small number of encoding bits. Furthermore, we propose an image reconstruction scheme for the proposed camera, utilizing both spike intervals and spike levels to recover the light intensity. We incorporate Mamba and propose a temporal-spatial selective scan mechanism to extract temporal-spatial correlation within spike streams. We employ a pyramid adaptive filtering and alignment module to achieve coarse-to-fine feature alignment. Experimental results show that the proposed scheme can achieve better imaging quality and outperform the existing spike camera in high dynamic range scenarios.Keywords
Funding Information
- National Natural Science Foundation of China (62072009, 22127807)
- Natural Science Foundation of Sichuan Province (2023NSFSC1972)
This publication has 42 references indexed in Scilit:
- VideoMamba: State Space Model for Efficient Video Understanding2024
- MambaIR: A Simple Baseline for Image Restoration with State-Space Model2024
- Uncertainty estimation in HDR imaging with Bayesian neural networksPattern Recognition, 2024
- 1000× Faster Camera and Machine Vision with Ordinary DevicesEngineering, 2022
- Learning to Reconstruct HDR Images from Events, with Applications to Depth and Flow PredictionInternational Journal of Computer Vision, 2021
- Learning to See in the Dark with Events2020
- Event Enhanced High-Quality Image Recovery2020
- Squeeze-and-Excitation Networks2018
- HDR-VQM: An objective quality measure for high dynamic range videoSignal Processing: Image Communication, 2015
- Multi-metric learning for multi-sensor fusion based classificationInformation Fusion, 2013