A Delay Metric for Video Object Detection: What Average Precision Fails to Tell
Authors:
- 1 October 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 573-582
Abstract
Average precision (AP) is a widely used metric to evaluate detection accuracy of image and video object detectors. In this paper, we analyze the object detection from video and point out that mAP alone is not sufficient to capture the temporal nature of video object detection. To tackle this problem, we propose a comprehensive metric, Average Delay (AD), to measure and compare detection delay. To facilitate delay evaluation, we carefully select a subset of ImageNet VID, which we name as ImageNet VIDT with an emphasis on complex trajectories. By extensively evaluating a wide range of detectors on VIDT, we show that most methods drastically increase the detection delay but still preserve mAP well. In other words, mAP is not sensitive enough to reflect the temporal characteristics of a video object detector. Our results suggest that video object detection methods should be evaluated with a delay metric, particularly for latency-critical applications such as autonomous vehicle perception.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- T-CNN: Tubelets With Convolutional Neural Networks for Object Detection From VideosIEEE Transactions on Circuits and Systems for Video Technology, 2017
- Deep Feature Flow for Video Recognition2017
- Multilayer and Multimodal Fusion of Deep Neural Networks for Video Classification2016
- Towards Real-Time Object Detection on Embedded SystemsIEEE Transactions on Emerging Topics in Computing, 2016
- Learning Activity Progression in LSTMs for Activity Detection and Early Detection2016
- Evaluating Real-Time Anomaly Detection Algorithms -- The Numenta Anomaly Benchmark2015
- A Discriminative Model with Multiple Temporal Scales for Action Prediction2014
- Vision meets robotics: The KITTI datasetThe International Journal of Robotics Research, 2013
- A large-scale benchmark dataset for event recognition in surveillance video2011
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009