DeepSample: DNN sampling-based testing for operational accuracy assessment
Open Access
- 12 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Abstract
Deep Neural Networks (DNN) are core components for classification and regression tasks of many software systems. Companies incur in high costs for testing DNN with datasets representative of the inputs expected in operation, as these need to be manually labelled. The challenge is to select a representative set of test inputs as small as possible to reduce the labelling cost, while sufficing to yield unbiased high-confidence estimates of the expected DNN accuracy. At the same time, testers are interested in exposing as many DNN mispredictions as possible to improve the DNN, ending up in the need for techniques pursuing a threefold aim: small dataset size, trustworthy estimates, mispredictions exposure. This study presents DeepSample, a family of DNN testing techniques for cost-effective accuracy assessment based on probabilistic sampling. We investigate whether, to what extent, and under which conditions probabilistic sampling can help to tackle the outlined challenge. We implement five new sampling-based testing techniques, and perform a comprehensive comparison of such techniques and of three further state-of-the-art techniques for both DNN classification and regression tasks. Results serve as guidance for best use of sampling-based testing for faithful and high-confidence estimates of DNN accuracy in operation at low cost.Keywords
All Related Versions
Funding Information
- Research Executive Agency (871342)
This publication has 30 references indexed in Scilit:
- Iterative Assessment and Improvement of DNN Operational Accuracy2023
- Assessing operational accuracy of CNN-based image classifiers using an oracle surrogateIntelligent Systems with Applications, 2023
- Guiding Deep Learning System Testing Using Surprise Adequacy2019
- Estimating confidence interval of software reliability with adaptive testing strategyJournal of Systems and Software, 2014
- An experimental study of adaptive testing for software reliability assessmentJournal of Systems and Software, 2008
- Optimal and adaptive testing for software reliability assessmentInformation and Software Technology, 2004
- A case study in cleanroom software engineering: the IBM COBOL Structuring Facility2003
- Software reliability-engineered testingComputer, 1996
- Engineering software under statistical quality controlIEEE Software, 1990
- A Generalization of Sampling Without Replacement from a Finite UniverseJournal of the American Statistical Association, 1952