Data-Driven Feature Tracking for Event Cameras
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5642-5651
Abstract
Because of their high temporal resolution, increased resilience to motion blur, and very sparse output, event cameras have been shown to be ideal for low-latency and low-bandwidth feature tracking, even in challenging scenarios. Existing feature tracking methods for event cameras are either handcrafted or derived from first principles but require extensive parameter tuning, are sensitive to noise, and do not generalize to different scenarios due to unmodeled effects. To tackle these deficiencies, we introduce the first data-driven feature tracker for event cameras, which leverages low-latency events to track features detected in a grayscale frame. We achieve robust performance via a novel frame attention module, which shares information across feature tracks. By directly transferring zero-shot from synthetic to real data, our data-driven tracker outperforms existing approaches in relative feature age by up to 120 % while also achieving the lowest latency. This performance gap is further increased to 130 % by adapting our tracker to real data with a novel self-supervision strategy. Multimedia Material A video is available at https://youtu.be/dtkXvNXcWRY and code at https://github.com/uzh-rpg/deep_ev_trackerKeywords
All Related Versions
Funding Information
- Swiss National Science Foundation
- National Centre of Competence in Research (NCCR) Robotics (51NF40_185543)
- European Research Council (ERC) (864042 (AGILEFLIGHT))
This publication has 31 references indexed in Scilit:
- Event-Based, 6-DOF Camera Tracking from Photometric Depth MapsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Event-based feature tracking with probabilistic data association2017
- The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAMThe International Journal of Robotics Research, 2017
- Low-latency visual odometry using event-based feature tracks2016
- Structure-from-Motion Revisited2016
- FlowNet: Learning Optical Flow with Convolutional Networks2015
- Direct Linear Transformation from Comparator Coordinates into Object Space Coordinates in Close-Range PhotogrammetryPhotogrammetric Engineering & Remote Sensing, 2015
- A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision SensorIEEE Journal of Solid-State Circuits, 2014
- A method for registration of 3-D shapesIEEE Transactions on Pattern Analysis and Machine Intelligence, 1992
- A Combined Corner and Edge Detector1988