EBBINNOT: A Hardware-Efficient Hybrid Event-Frame Tracker for Stationary Dynamic Vision Sensors
- 26 May 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 9 (21) , 20902-20917
Abstract
As an alternative sensing paradigm, dynamic vision sensors (DVS) have been recently explored to tackle scenarios where conventional sensors result in high data rate and processing time. This paper presents a hybrid event-frame approach for detecting and tracking objects recorded by a stationary neuromorphic sensor, thereby exploiting the sparse DVS output in a low-power setting for traffic monitoring. Specifically, we propose a hardware efficient processing pipeline that optimizes memory and computational needs that enable long-term battery powered usage for IoT applications. To exploit the background removal property of a static DVS, we propose an event-based binary image creation that signals presence or absence of events in a frame duration. This reduces memory requirement and enables usage of simple algorithms like median filtering and connected component labeling for denoise and region proposal respectively. To overcome the fragmentation issue, a YOLO inspired neural network based detector and classifier to merge fragmented region proposals has been proposed. Finally, a new overlap based tracker was implemented, exploiting overlap between detections and tracks is proposed with heuristics to overcome occlusion. The proposed pipeline is evaluated with more than 5 hours of traffic recording spanning three different locations on two different neuromorphic sensors (DVS and CeleX) and demonstrate similar performance. Compared to existing event-based feature trackers, our method provides similar accuracy while needing. ≈ 6 × less computes. To the best of our knowledge, this is the first time a stationary DVS based traffic monitoring solution is extensively compared to simultaneously recorded RGB frame-based methods while showing tremendous promise by outperforming state-of-the-art deep learning solutions. The traffic dataset is publicly made available at: https://nusneuromorphic.github.io/dataset/index.html.Keywords
All Related Versions
This publication has 68 references indexed in Scilit:
- Visual Tracking Using Neuromorphic Asynchronous Event-Based CamerasNeural Computation, 2015
- Asynchronous Event-Based Multikernel Algorithm for High-Speed Visual Features TrackingIEEE Transactions on Neural Networks and Learning Systems, 2014
- A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision SensorIEEE Journal of Solid-State Circuits, 2014
- Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking OutputProceedings of the IEEE, 2014
- A million spiking-neuron integrated circuit with a scalable communication network and interfaceScience, 2014
- Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensorFrontiers in Neuroscience, 2013
- Asynchronous event‐based high speed vision for microparticle trackingJournal of Microscopy, 2011
- A 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision SensorIEEE Journal of Solid-State Circuits, 2008
- Automatic Traffic Surveillance System for Vehicle Tracking and ClassificationIEEE Transactions on Intelligent Transportation Systems, 2006
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998