Towards more reliable evaluation in pedestrian detection by rethinking “ignore regions”
Open Access
- 22 February 2024
- journal article
- research article
- Published by Springer Nature in Visual Intelligence
- Vol. 2 (1) , 1-13
Abstract
It remains a challenging task to detect pedestrians in crowds and it needs more efforts to understand why the detectors fail. When we perform an error analysis based on the traditional evaluation strategy, we find that it produces many misleading false positives, which in fact cover occluded pedestrians. The reason for this is that we usually have two kinds of annotations in the dataset: regular pedestrians (detection targets) labeled by full-body boxes and ignored pedestrians (NOT detection targets) labeled by visible boxes. Ignored pedestrians are labeled as an additional category termed the “ignore region”. Nevertheless, our detectors always predict a full-body box for each pedestrian. This gap results in the following case: when a detector successfully predicts a full-body box for those ignored pedestrians, a false positive is triggered due to the low overlap between the predicted full-body box and the labeled visible box for the ignored pedestrian. This becomes even more harmful as the detector improves and becomes more capable of locating occluded pedestrians. To alleviate this issue, we devise a new pedestrian detection pipeline, which considers the additional visible box at both the detection and evaluation stages. During detection, we predict an extra visible box apart from the full-body box for every instance; during evaluation, we employ visible boxes instead of full-body boxes to match the “ignore region”. We apply the new pipeline to dozens of detection methods and validate the effectiveness of our pipeline in reducing the over-reporting of false positives and providing more reliable evaluation results.Keywords
Funding Information
- National Natural Science Foundation of China (62172225)
- Fundamental Research Funds for the Central Universities (30920032201)
This publication has 41 references indexed in Scilit:
- VarifocalNet: An IoU-aware Dense Object Detector2021
- End-to-End Object Detection with Transformers2020
- Learning Hierarchical Graph for Occluded Pedestrian Detection2020
- PEN: Pose-Embedding Network for Pedestrian DetectionIEEE Transactions on Circuits and Systems for Video Technology, 2020
- Human Detection Aided by Deeply Learned Semantic MasksIEEE Transactions on Circuits and Systems for Video Technology, 2019
- S3D: Scalable Pedestrian Detection via Score Scale Surface DiscriminationIEEE Transactions on Circuits and Systems for Video Technology, 2019
- Multi-Grained Deep Feature Learning for Robust Pedestrian DetectionIEEE Transactions on Circuits and Systems for Video Technology, 2018
- Person Re-identification with Deep Similarity-Guided Graph Neural Network2018
- Bi-box Regression for Pedestrian Detection and Occlusion Estimation2018
- Pedestrian Detection: An Evaluation of the State of the ArtIEEE Transactions on Pattern Analysis and Machine Intelligence, 2011