Model vs system level testing of autonomous driving systems: a replication and extension study
Open Access
- 2 May 2023
- journal article
- research article
- Published by Springer Nature in Empirical Software Engineering
- Vol. 28 (3) , 1-26
Abstract
Offline model-level testing of autonomous driving software is much cheaper, faster, and diversified than in-field, online system-level testing. Hence, researchers have compared empirically model-level vs system-level testing using driving simulators. They reported the general usefulness of simulators at reproducing the same conditions experienced in-field, but also some inadequacy of model-level testing at exposing failures that are observable only in online mode. In this work, we replicate the reference study on model vs system-level testing of autonomous vehicles while acknowledging several assumptions that we had reconsidered. These assumptions are related to several threats to validity affecting the original study that motivated additional analysis and the development of techniques to mitigate them. Moreover, we also extend the replicated study by evaluating the original findings when considering a physical, radio-controlled autonomous vehicle. Our results show that simulator-based testing of autonomous driving systems yields predictions that are close to the ones of real-world datasets when using neural-based translation to mitigate the reality gap induced by the simulation platform. On the other hand, model-level testing failures are in line with those experienced at the system level, both in simulated and physical environments, when considering the pre-failure site, similar-looking images, and accurate labels.Keywords
Funding Information
- Università Della Svizzera Italiana
This publication has 50 references indexed in Scilit:
- Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehiclesJournal of Systems and Software, 2018
- DeepXplore2017
- Can Robot Navigation Bugs Be Found in Simulation? An Exploratory Study2017
- Search-based test case generation for Cyber-Physical Systems2017
- Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?Transportation Research Part A: Policy and Practice, 2016
- Testing Autonomous and Highly Configurable Systems: Challenges and Feasible Solutions2016
- Testing advanced driver assistance systems using multi-objective search and neural networks2016
- Testing of Advanced Driver Assistance Towards Automated Driving: A Survey and Taxonomy on Existing Approaches and Open Questions2015
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Individual Comparisons by Ranking MethodsBiometrics Bulletin, 1945