An Empirical Study on Code Search Pre-trained Models: Academic Progresses vs. Industry Requirements
Open Access
- 24 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 41-50
Abstract
With the rapid development of pre-trained source code models, code search has made fundamental advances. However, a thorough evaluation of the effectiveness of academic code search models in addressing the need of the industry is overlooked. We propose to conduct a ground-breaking evaluation of existing code search models w.r.t. their adaptability, scalability, robustness, and semantic sensitivity. First, we evaluate the influence of the queries’ semantic attributes on searching performance extensively, and design strategies to reduce the impact of the incomplete semantics. Then, we use variants of queries to test the model’s adaptability and robustness. Next, we classify queries to different search purposes to determine the cross-type searching applicability of the models. Finally, we measure the effects of multilingual efficient fine-tuning on the model performance, and provide a reliable way to reduce the costs of developing and deploying code search systems for the industry. These contribute well to narrowing the gap between academic progress and industry requirements of code search.Keywords
Funding Information
- CCF-Huawei Populus Grove Fund
- Huawei-NJU Creative Laboratory for the Next Programming
This publication has 41 references indexed in Scilit:
- On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot2023
- One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization2023
- An Empirical Comparison of Pre-Trained Models of Source Code2023
- CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models2023
- What do pre-trained code models know about code?2021
- Assessing Generalizability of CodeBERT2021
- Multi-modal Attention Network Learning for Semantic Source Code Retrieval2019
- Combining Word2Vec with Revised Vector Space Model for Better Code Retrieval2017
- A search log mining based query expansion technique to improve effectiveness in code search2016
- Improving source code search with natural language phrasal representations of method signatures2011