Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Preprint
Abstract
Deep Neural Networks (DNNs) have been widely used to perform real-world tasks in cyber-physical systems such as Autonomous Driving Systems (ADS). Ensuring the correct behavior of such DNN-Enabled Systems (DES) is a crucial topic. Online testing is one of the promising modes for testing such systems with their application environments (simulated or real) in a closed loop taking into account the continuous interaction between the systems and their environments. However, the environmental variables (e.g., lighting conditions) that might change during the systems' operation in the real world, causing the DES to violate requirements (safety, functional), are often kept constant during the execution of an online test scenario due to the two major challenges: (1) the space of all possible scenarios to explore would become even larger if they changed and (2) there are typically many requirements to test simultaneously. In this paper, we present MORLOT (Many-Objective Reinforcement Learning for Online Testing), a novel online testing approach to address these challenges by combining Reinforcement Learning (RL) and many-objective search. MORLOT leverages RL to incrementally generate sequences of environmental changes while relying on many-objective search to determine the changes so that they are more likely to achieve any of the uncovered objectives. We empirically evaluate MORLOT using CARLA, a high-fidelity simulator widely used for autonomous driving research, integrated with Transfuser, a DNN-enabled ADS for end-to-end driving. The evaluation results show that MORLOT is significantly more effective and efficient than alternatives with a large effect size. In other words, MORLOT is a good option to test DES with dynamically changing environments while accounting for multiple safety requirements.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their CollisionsIEEE Transactions on Software Engineering, 2022
- Can Offline Testing of Deep Neural Networks Replace Their Online Testing?Empirical Software Engineering, 2021
- Model-based exploration of the frontier of behaviours for deep learning system testing2020
- Machine Learning Testing: Survey, Landscapes and HorizonsIEEE Transactions on Software Engineering, 2020
- Automatically testing self-driving cars with search-based procedural content generation2019
- A general reinforcement learning algorithm that masters chess, shogi, and Go through self-playScience, 2018
- Deep Reinforcement Learning: A Brief SurveyIEEE Signal Processing Magazine, 2017
- An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box ConstraintsIEEE Transactions on Evolutionary Computation, 2013
- Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech RecognitionIEEE Transactions on Audio, Speech, and Language Processing, 2011
- MOEA/D: A Multiobjective Evolutionary Algorithm Based on DecompositionIEEE Transactions on Evolutionary Computation, 2007