Seeing into Darkness: Scotopic Visual Recognition
Preprint
Abstract
Images are formed by counting how many photons traveling from a given set of directions hit an image sensor during a given time interval. When photons are few and far in between, the concept of `image' breaks down and it is best to consider directly the flow of photons. Computer vision in this regime, which we call `scotopic', is radically different from the classical image-based paradigm in that visual computations (classification, control, search) have to take place while the stream of photons is captured and decisions may be taken as soon as enough information is available. The scotopic regime is important for biomedical imaging, security, astronomy and many other fields. Here we develop a framework that allows a machine to classify objects with as few photons as possible, while maintaining the error rate below an acceptable threshold. A dynamic and asymptotically optimal speed-accuracy tradeoff is a key feature of this framework. We propose and study an algorithm to optimize the tradeoff of a convolutional network directly from lowlight images and evaluate on simulated images from standard datasets. Surprisingly, scotopic systems can achieve comparable classification performance as traditional vision systems while using less than 0.1% of the photons in a conventional image. In addition, we demonstrate that our algorithms work even when the illuminance of the environment is unknown and varying. Last, we outline a spiking neural network coupled with photon-counting sensors as a power-efficient hardware realization of scotopic algorithms.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Imaging with a small number of photonsNature Communications, 2015
- A Novel Connectionist System for Unconstrained Handwriting RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2008
- Principles and features of single-photon avalanche diode arraysSensors and Actuators A: Physical, 2007
- The physics of optimal decision making: A formal analysis of models of performance in two-alternative forced-choice tasks.Psychological Review, 2006
- Approximation of dynamical time-variant systems by continuous-time recurrent neural networksIEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, 2005
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Speed of processing in the human visual systemNature, 1996
- Silicon retina with correlation-based, velocity-tuned pixelsIEEE Transactions on Neural Networks, 1993
- Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in positionBiological Cybernetics, 1980
- Intensity of light and speed of vision: I.Journal of Experimental Psychology, 1929