How certain are we that our automated driving system is safe?
Open Access
- 21 April 2023
- journal article
- research article
- Published by Taylor & Francis in Traffic Injury Prevention
- Vol. 24 (sup1) , S131-S140
Abstract
Objective: Regulations are currently being drafted by the European Commission for the safe introduction of automated driving systems (ADSs) with conditional or higher automation (SAE level 3 and above). One of the main challenges for complying with the drafted regulations is proving that the residual risk of an ADS is lower than the existing state of the art without the ADS and that the current safety state of European roads is not compromised. Therefore, much research has been conducted to estimate the safety risk of ADS. One proposed method for estimating the risk is data-driven, scenario-based assessment, where tests are partially automatically generated based on recorded traffic data. Although this is a promising method, uncertainties in the estimated risk arise from, among others, the limited number of tests that are conducted and the limited data that have been used to generate the tests. This work addresses the following question: "Given the limitations of the data and the number of tests, what is the uncertainty of the estimated safety risk of the ADS?" Methods: To compute the safety risk, parameterized test scenarios are based on large-scale collections of road scenarios that are stored in a scenario database. The exposure of the scenarios and the parameter distributions are estimated using the data as well as confidence bounds of these estimates. Next, virtual simulations are conducted of the scenarios for a variety of parameter values. Using a probabilistic framework, all results are combined to estimate the residual risk as well as the uncertainty of this estimation. Results: The results are used to provide confidence bounds on the calculated fatality rate in case an ADS is implemented in the vehicle. For example, using the proposed probabilistic framework, it is possible to claim with 95% certainty that the fatality rate is less than 10(-7) fatalities per hour of driving. The proposed method is illustrated with a case study in which the risk and its uncertainty are quantified for a longitudinal controller in 3 different types of scenarios. The case study code is publicly available. Conclusions: If results show that the uncertainty is too high, the proposed method allows answering questions like "How much more data do we need?" or "How many more (virtual) simulations must be conducted?" Therefore, the method can be used to set requirements on the amount of data and the number of (virtual) simulations. For a reliable risk estimate, though, much more data are needed than those used in the case study. Furthermore, because the method relies on (virtual) simulations, the reliability of the result depends on the validity of the models used in the simulations. The presented case study illustrates that the proposed method is able to quantify the uncertainty of the estimated safety risk of an ADS. Future work involves incorporating the proposed method into the type approval framework for future ADSs of SAE levels 3, 4, and 5, as proposed in the upcoming European Union implementing regulation for ADS.Keywords
Funding Information
- European Union’s Horizon 2020 Research and Innovation Programme (101006664)
This publication has 31 references indexed in Scilit:
- Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental dataTransportation Research Part C: Emerging Technologies, 2014
- ks: Kernel Density Estimation and Kernel Discriminant Analysis for Multivariate Data inRJournal of Statistical Software, 2007
- Kernel estimation of density level setsJournal of Multivariate Analysis, 2006
- Robustness of the Ljung-Box Test and its Rank EquivalentSSRN Electronic Journal, 2003
- Nonparametric Importance SamplingJournal of the American Statistical Association, 1996
- Bootstrap Methods: Another Look at the JackknifeThe Annals of Statistics, 1979
- On a measure of lack of fit in time series modelsBiometrika, 1978
- On Estimation of a Probability Density Function and ModeThe Annals of Mathematical Statistics, 1962
- On the Exact Variance of ProductsJournal of the American Statistical Association, 1960
- Remarks on Some Nonparametric Estimates of a Density FunctionThe Annals of Mathematical Statistics, 1956