Improving Deep Learning Library Testing with Machine Learning
Preprint
Abstract
Deep Learning (DL) libraries like TensorFlow and Pytorch simplify machine learning (ML) model development but are prone to bugs due to their complex design. Bug-finding techniques exist, but without precise API specifications, they produce many false alarms. Existing methods to mine API specifications lack accuracy. We explore using ML classifiers to determine input validity. We hypothesize that tensor shapes are a precise abstraction to encode concrete inputs and capture relationships of the data. Shape abstraction severely reduces problem dimensionality, which is important to facilitate ML training. Labeled data are obtained by observing runtime outcomes on a sample of inputs and classifiers are trained on sets of labeled inputs to capture API constraints. Our evaluation, conducted over 183 APIs from TensorFlow and Pytorch, shows that the classifiers generalize well on unseen data with over 91% accuracy. Integrating these classifiers into the pipeline of ACETest, a SoTA bug-finding technique, improves its pass rate from ~29% to ~61%. Our findings suggest that ML-enhanced input classification is an important aid to scale DL library testing.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Learning to Prune Infeasible Paths in Generalized Symbolic Execution2022
- The symptoms, causes, and repairs of bugs inside a deep learning libraryJournal of Systems and Software, 2021
- Audee2020
- Deep learning library testing via effective model generation2020
- A study of learning likely data structure properties using machine learning modelsInternational Journal on Software Tools for Technology Transfer, 2020
- Taxonomy of real faults in deep learning systems2020
- CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries2019
- Training Binary Classifiers as Data Structure Invariants2019
- XGBoost2016
- Finding latent code errors via machine learning over program executions2004