Reinforcement Learning for Online Testing of Autonomous Driving Systems: a Replication and Extension Study
Preprint
Abstract
In a recent study, Reinforcement Learning (RL) used in combination with many-objective search, has been shown to outperform alternative techniques (random search and many-objective search) for online testing of Deep Neural Network-enabled systems. The empirical evaluation of these techniques was conducted on a state-of-the-art Autonomous Driving System (ADS). This work is a replication and extension of that empirical study. Our replication shows that RL does not outperform pure random test generation in a comparison conducted under the same settings of the original study, but with no confounding factor coming from the way collisions are measured. Our extension aims at eliminating some of the possible reasons for the poor performance of RL observed in our replication: (1) the presence of reward components providing contrasting or useless feedback to the RL agent; (2) the usage of an RL algorithm (Q-learning) which requires discretization of an intrinsically continuous state space. Results show that our new RL agent is able to converge to an effective policy that outperforms random testing. Results also highlight other possible improvements, which open to further investigations on how to best leverage RL for online ADS testing.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems2023
- Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies2022
- Multi-Modal Fusion Transformer for End-to-End Autonomous Driving2021
- AV-FUZZER: Finding Safety Violations in Autonomous Driving Systems2020
- Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validatio2019
- Adaptive stress testing of airborne collision avoidance systems2015
- Replication of empirical studies in software engineering research: a systematic mapping studyEmpirical Software Engineering, 2012
- The Design of Replicated StudiesThe American Statistician, 1993
- Multiple Comparisons Using Rank SumsTechnometrics, 1964
- The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of VarianceJournal of the American Statistical Association, 1937