Quantifying the LiDAR Sim-to-Real Domain Shift: A Detailed Investigation Using Object Detectors and Analyzing Point Clouds at Target-Level
Authors:
- 2 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 8 (4) , 2970-2982
Abstract
LiDAR object detection algorithms based on neural networks for autonomous driving require large amounts of data for training, validation, and testing. As real-world data collection and labeling are time-consuming and expensive, simulation-based synthetic data generation is a viable alternative. However, using simulated data for the training of neural networks leads to a domain shift of training and testing data due to differences in scenes, scenarios, and distributions. In this work, we quantify the sim-to-real domain shift by means of LiDAR object detectors trained with a new scenario-identical real-world and simulated dataset. In addition, we answer the questions of how well the simulated data resembles the real-world data and how well object detectors trained on simulated data perform on real-world data. Further, we analyze point clouds at the target-level by comparing real-world and simulated point clouds within the 3D bounding boxes of the targets. Our experiments show that a significant sim-to-real domain shift exists even for our scenario-identical datasets. This domain shift amounts to an average precision reduction of around 14% for object detectors trained with simulated data. Additional experiments reveal that this domain shift can be lowered by introducing a simple noise model in simulation. We further show that a simple downsampling method to model real-world physics does not influence the performance of the object detectors.Keywords
All Related Versions
Funding Information
- Federal Ministry of Education and Research of Germany
- BMBF
- Institute for Automotive Technology
This publication has 26 references indexed in Scilit:
- Robustness and Deployability of Deep Object Detectors in Autonomous Driving2019
- Performance of LiDAR object detection deep learning architectures based on artificially generated point cloud data from CARLA simulator2019
- Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data2019
- A LiDAR Point Cloud Generator2018
- Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization2018
- Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving ScenesInternational Journal of Computer Vision, 2018
- VirtualWorlds as Proxy for Multi-object Tracking Analysis2016
- Understanding how image quality affects deep neural networks2016
- Vision meets robotics: The KITTI datasetThe International Journal of Robotics Research, 2013
- A theory of learning from different domainsMachine Learning, 2009