Localization is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix it
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 22150-22159
Abstract
The task of online mapping is to predict a local map using current sensor observations, e.g. from lidar and camera, without relying on a pre-built map. State-of-the-art methods are based on supervised learning and are trained predominantly using two datasets: nuScenes and Argoverse 2. However, these datasets revisit the same geographic locations across training, validation, and test sets. Specifically, over 80% of nuScenes and 40% of Argoverse 2 validation and test samples are less than 5 m from a training sample. At test time, the methods are thus evaluated more on how well they localize within a memorized implicit map built from the training data than on extrapolating to unseen locations. Naturally, this data leakage causes inflated performance numbers and we propose geographically disjoint data splits to reveal the true performance in unseen environments. Experimental results show that methods perform considerably worse, some dropping more than 45 mAp, when trained and evaluated on proper data splits. Additionally, a reassessment of prior design choices reveals diverging conclusions from those based on the original split. Notably, the impact of lifting methods and the support from auxiliary tasks (e.g., depth supervision) on performance appears less substantial or follows a different trajectory than previously perceived. https://github.com/LiljaAdam/geographical-splitsKeywords
All Related Versions
Funding Information
- Swedish Research Council (2022–06725)
This publication has 28 references indexed in Scilit:
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- ST-P3: End-to-End Vision-Based Autonomous Driving via Spatial-Temporal Feature Learning2022
- End-to-End Object Detection with Transformers2020
- Spatial validation reveals poor predictive performance of large-scale ecological mapping modelsNature Communications, 2020
- Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks2020
- nuScenes: A Multimodal Dataset for Autonomous Driving2020
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning2020
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset2020
- Argoverse: 3D Tracking and Forecasting With Rich Maps2019
- Inverse perspective mapping simplifies optical flow computation and obstacle detectionBiological Cybernetics, 1991