Towards Exploring the Limitations of Test Selection Techniques on Graph Neural Networks: An Empirical Study
Open Access
- 22 July 2024
- journal article
- research article
- Published by Springer Nature in Empirical Software Engineering
- Vol. 29 (5) , 1-41
Abstract
Graph Neural Networks (GNNs) have gained prominence in various domains, such as social network analysis, recommendation systems, and drug discovery, due to their ability to model complex relationships in graph-structured data. GNNs can exhibit incorrect behavior, resulting in severe consequences. Therefore, testing is necessary and pivotal. However, labeling all test inputs for GNNs can be prohibitively costly and time-consuming, especially when dealing with large and complex graphs. In response to these challenges, test selection has emerged as a strategic approach to alleviate labeling expenses. The objective of test selection is to select a subset of tests from the complete test set. While various test selection techniques have been proposed for traditional deep neural networks (DNNs), their adaptation to GNNs presents unique challenges due to the distinctions between DNN and GNN test data. Specifically, DNN test inputs are independent of each other, whereas GNN test inputs (nodes) exhibit intricate interdependencies. Therefore, it remains unclear whether DNN test selection approaches can perform effectively on GNNs. To fill the gap, we conduct an empirical study that systematically evaluates the effectiveness of various test selection methods in the context of GNNs, focusing on three critical aspects: 1) Misclassification detection: selecting test inputs that are more likely to be misclassified; 2) Accuracy estimation: selecting a small set of tests to precisely estimate the accuracy of the whole testing set; 3) Performance enhancement: selecting retraining inputs to improve the GNN accuracy. Our empirical study encompasses 7 graph datasets and 8 GNN models, evaluating 22 test selection approaches. Our study includes not only node classification datasets but also graph classification datasets. Our findings reveal that: 1) In GNN misclassification detection, confidence-based test selection methods, which perform well in DNNs, do not demonstrate the same level of effectiveness; 2) In terms of GNN accuracy estimation, clustering-based methods, while consistently performing better than random selection, provide only slight improvements; 3) Regarding selecting inputs for GNN performance improvement, test selection methods, such as confidence-based and clustering-based test selection methods, demonstrate only slight effectiveness; 4) Concerning performance enhancement, node importance-based test selection methods are not suitable, and in many cases, they even perform worse than random selection.Keywords
Funding Information
- Fonds National de la Recherche Luxembourg (AFR PHD 17036341)
This publication has 67 references indexed in Scilit:
- Object detection by crossing relational reasoning based on graph neural networkMachine Vision and Applications, 2021
- Molecular generative Graph Neural Networks for Drug DiscoveryNeurocomputing, 2021
- Can Offline Testing of Deep Neural Networks Replace Their Online Testing?Empirical Software Engineering, 2021
- A compact review of molecular property prediction with graph neural networksDrug Discovery Today: Technologies, 2020
- Graph neural networks: A review of methods and applicationsAI Open, 2020
- A novel method to evaluate node importance in complex networksPhysica A: Statistical Mechanics and its Applications, 2019
- Finding better active learners for faster literature reviewsEmpirical Software Engineering, 2018
- Propagation kernels: efficient graph kernels from propagated informationMachine Learning, 2015
- Identifying node importance in complex networksPhysica A: Statistical Mechanics and its Applications, 2015
- IAM Graph Database Repository for Graph Based Pattern Recognition and Machine Learning2008