Coding-PTMs: How to Find Optimal Code Pre-trained Models for Code Embedding in Vulnerability Detection?
Open Access
- 27 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1732-1744
Abstract
Vulnerability detection is garnering increasing attention in software engineering, since code vulnerabilities possibly pose significant security. Recently, reusing various code pre-trained models (e.g., CodeBERT, CodeT5, and CodeGen) has become common for code embedding without providing reasonable justifications in vulnerability detection. The premise for casually utilizing pre-trained models (PTMs) is that the code embeddings generated by different PTMs would generate a similar impact on the performance. Is that TRUE? To answer this important question, we systematically investigate the effects of code embedding generated by ten different code PTMs on the performance of vulnerability detection, and get the answer, i.e., that is NOT true. We observe that code embedding generated by various code PTMs can indeed influence the performance and selecting an embedding technique based on parameter scales and embedding dimension is not reliable. Our findings highlight the necessity of quantifying and evaluating the characteristics of code embedding generated by various code PTMs to understand the effects. To achieve this goal, we analyze the numerical representation and data distribution of code embedding generated by different PTMs to evaluate differences and characteristics. Based on these insights, we propose Coding-PTMs, a recommendation framework to assist engineers in selecting optimal code PTMs for their specific vulnerability detection tasks. Specifically, we define thirteen code embedding metrics across three dimensions (i.e., statistics, norm, and distribution) for constructing a specialized code PTM recommendation dataset. We then employ a Random Forest classifier to train a recommendation model and identify the optimal code PTMs from the candidate model zoo. We encourage engineers to use our Coding-PTMs to evaluate the characteristics of code embeddings generated by candidate code PTMs on the performance and recommend optimal code PTMs for code embedding in their vulnerability detection tasks.Keywords
Funding Information
- National Natural Science Foundation of China (U2241216, 62202223)
- Natural Science Foundation of Jiangsu Province (BK20220881)
- Foundation of the Key National Laboratory of New Technology in Computer Software (Nanjing University) (KFKT2022B20)
- Fundamental Research Funds for the Central Universities (NJ2024029)
This publication has 49 references indexed in Scilit:
- DeepVulSeeker: A novel vulnerability identification framework via code graph structure and pre-training mechanismFuture Generation Computer Systems, 2023
- CSGVD: A deep learning approach combining sequence and graph embedding for source code vulnerability detectionJournal of Systems and Software, 2023
- ASSBert: Active and semi-supervised bert for smart contract vulnerability detectionJournal of Information Security and Applications, 2023
- Can pre-trained code embeddings improve model performance? Revisiting the use of code embeddings in software engineering tasksEmpirical Software Engineering, 2022
- An empirical evaluation of deep learning‐based source code vulnerability detection: Representation versus modelsJournal of Software: Evolution and Process, 2022
- The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approachesJournal of Network and Computer Applications, 2021
- The impact factors on the performance of machine learning-based vulnerability detection: A comparative studyJournal of Systems and Software, 2020
- The impact of automated feature selection techniques on the interpretation of defect modelsEmpirical Software Engineering, 2020
- An empirical assessment of best-answer prediction models in technical Q&A sitesEmpirical Software Engineering, 2018
- An empirical study on software defect prediction with a simplified metric setInformation and Software Technology, 2015