Deep learning library testing via effective model generation
Authors:
Top Cited Papers
Open Access
- 8 November 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 788-799
Abstract
Deep learning (DL) techniques are rapidly developed and have been widely adopted in practice. However, similar to traditional software systems, DL systems also contain bugs, which could cause serious impacts especially in safety-critical domains. Recently, many research approaches have focused on testing DL models, while little attention has been paid for testing DL libraries, which is the basis of building DL models and directly affects the behavior of DL systems. In this work, we propose a novel approach, LEMON, to testing DL libraries. In particular, we (1) design a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors; and (2) propose a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs, so as to mitigate the impact of potential noise introduced by uncertain factors in DL libraries. We conducted an empirical study to evaluate the effectiveness of LEMON with 20 release versions of 4 widely-used DL libraries, i.e., TensorFlow, Theano, CNTK, MXNet. The results demonstrate that LEMON detected 24 new bugs in the latest release versions of these libraries, where 7 bugs have been confirmed and one bug has been fixed by developers. Besides, the results confirm that the heuristic strategy for model generation indeed effectively guides LEMON in amplifying the inconsistent degrees for bugs.Keywords
Funding Information
- National Natural Science Foundation of China (61872263,U1836214,61802275)
- Intelligent Manufacturing Special Fund of Tianjin (20191012,20193155)
This publication has 40 references indexed in Scilit:
- Policy compression for aircraft collision avoidance systems2016
- Supporting oracle construction via static analysis2016
- A Survey on Software Fault LocalizationIEEE Transactions on Software Engineering, 2016
- Threats to the validity of mutation-based test assessment2016
- DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving2015
- Automated testing for SQL injection vulnerabilities: an input mutation approach2014
- Speech recognition with deep recurrent neural networks2013
- Roulette-wheel selection via stochastic acceptancePhysica A: Statistical Mechanics and its Applications, 2012
- Higher Order Mutation TestingInformation and Software Technology, 2009
- What every computer scientist should know about floating-point arithmeticACM Computing Surveys, 1991