An Empirical Study on Noisy Label Learning for Program Understanding
Open Access
- 12 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Abstract
Recently, deep learning models have been widely applied in program understanding tasks, and these models achieve state-of-the-art results on many benchmark datasets. A major challenge of deep learning for program understanding is that the effectiveness of these approaches depends on the quality of their datasets, and these datasets often contain noisy data samples. A typical kind of noise in program understanding datasets is label noise, which means that the target outputs for some inputs are incorrect. Researchers have proposed various approaches to alleviate the negative impact of noisy labels, and formed a new research topic: noisy label learning (NLL). In this paper, we conduct an empirical study on the effectiveness of noisy label learning on deep learning for program understanding datasets. We evaluate various NLL approaches and deep learning models on three tasks: program classification, vulnerability detection, and code summarization. From the evaluation results, we come to the following findings: 1) small trained-from-scratch models are prone to label noises in program understanding, while large pre-trained models are highly robust against them. 2) NLL approaches significantly improve the program classification accuracies for small models on noisy training sets, but they only slightly benefit large pre-trained models in classification accuracies. 3) NLL can effectively detect synthetic noises in program understanding, but struggle in detecting real-world noises. We believe our findings can provide insights on the abilities of NLL in program understanding, and shed light on future works in tackling noises in software engineering datasets. We have released our code at https://github.com/jacobwwh/noise_SE.Keywords
All Related Versions
Funding Information
- National Research Foundation, Singapore, and the Cyber Security Agency under its National Cybersecurity R&D Programme (NCRP25-P04-TAICeN)
- NRF Investigatorship (NRF-NRFI06-2020-0001)
This publication has 30 references indexed in Scilit:
- Data Preparation for Software Vulnerability Prediction: A Systematic Literature ReviewIEEE Transactions on Software Engineering, 2022
- Learning From Noisy Labels With Deep Neural Networks: A SurveyIEEE Transactions on Neural Networks and Learning Systems, 2022
- Privacy-Preserving Efficient Federated-Learning Model DebuggingIEEE Transactions on Parallel and Distributed Systems, 2021
- Sample-level Data Selection for Federated Learning2021
- Efficient Federated-Learning Model Debugging2021
- Detecting Code Clones with Graph Neural Network and Flow-Augmented Abstract Syntax Tree2020
- A Novel Neural Source Code Representation Based on Abstract Syntax Tree2019
- Deep code comment generation2018
- It's not a bug, it's a feature: How misclassification impacts bug prediction2013
- Learning From Noisy ExamplesMachine Learning, 1988