CIT4DNN: Generating Diverse and Rare Inputs for Neural Networks Using Latent Space Combinatorial Testing
Open Access
- 12 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Abstract
Deep neural networks (DNN) are being used in a wide range of applications including safety-critical systems. Several DNN test generation approaches have been proposed to generate fault-revealing test inputs. However, the existing test generation approaches do not systematically cover the input data distribution to test DNNs with diverse inputs, and none of the approaches investigate the relationship between rare inputs and faults. We propose cit4dnn, an automated black-box approach to generate DNN test sets that are feature-diverse and that comprise rare inputs. cit4dnn constructs diverse test sets by applying combinatorial interaction testing to the latent space of generative models and formulates constraints over the geometry of the latent space to generate rare and fault-revealing test inputs. Evaluation on a range of datasets and models shows that cit4dnn generated tests are more feature diverse than the state-of-the-art, and can target rare fault-revealing testing inputs more effectively than existing methods.Keywords
Funding Information
- NSF (2019239, 2129824, 2217071)
- Air Force Office of Scientific Research (FA9550-21-0164)
- DARPA AIE Geometries of Learning Program (HR0011229007)
- Lockheed Martin Advanced Technology Laboratories
This publication has 43 references indexed in Scilit:
- Medical image segmentation using deep learning: A surveyIET Image Processing, 2022
- DeepReach: A Deep Learning Approach to High-Dimensional Reachability2021
- Validation of Image-Based Neural Network Controllers through Adaptive Stress Testing2020
- Advances in Combinatorial Testing2019
- DeepTest2018
- DeepXplore2017
- Combinatorial Coverage Measurement Concepts and Applications2013
- Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy ApproachIEEE Transactions on Software Engineering, 2008
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- The AETG system: an approach to testing based on combinatorial designIEEE Transactions on Software Engineering, 1997