Weighted Suspiciousness and Balanced Aggregation to Boost Spectrum-based Fault Localization of Deep Learning Models

Deep learning (DL) models have proven to be highly successful and are now essential to our everyday routines. However, DL models, like traditional software, inevitably contain bugs that affect their performance in real-world scenarios. Effective software engineering techniques are necessary to ensure their dependability. In recent years, fault localization methods for DL models have gained significant attention as a valuable tool for improving the reliability of DL models. Owing to the data-driven programming paradigm, traditional fault localization techniques are challenging to apply directly to DL programs. Previous studies have shown that neuron errors within models can lead to abnormal behavior, and they fix the DL model errors from the perspective of neurons. Nonetheless, there remains a significant gap between the DL program statement and model errors. To tackle this problem, this paper proposes a novel fault localization method for DL models, named wei G hted s U sp I ciousness an D balanc E d agg R egation ( \(\mathsf{GUIDER}\) ) that revisits the idea and challenge of spectrum-based fault localization in the context of DL models. For pre-trained DL models, \(\mathsf{GUIDER}\) utilizes neuron coverage information and test case confidence to compute weighted neuron suspiciousness values and employs balanced aggregation methods to elevate these values from the neuron level to the layer level, which establishes a bridge between the DL model and the DL program, facilitating the developers’ debugging process. We evaluate \(\mathsf{GUIDER}\) using 161 real model bugs collected from StackOverflow and five state-of-the-art fault localization methods for DL models as baselines. The results indicate that (a) our method successfully localizes 67% of the model bugs by ranking the buggy layer to the first place (i.e., top- \(1\) ), significantly outperforming all five baselines, and (b) our method maintains an acceptable time overhead compared with all baseline methods.