Enhancing Discriminative Tasks by Guiding the Pre-trained Language Model with Large Language Model's Experience
Preprint
Abstract
Large Language Models (LLMs) and pre-trained Language Models (LMs) have achieved impressive success on many software engineering tasks (e.g., code completion and code generation). By leveraging huge existing code corpora (e.g., GitHub), these models aim to understand the patterns in source code and use these patterns to predict code properties. However, fine-tuning LLMs is time-consuming and costly for end users and small organizations. Furthermore, fine-tuning LMs heavily depends on the amount and quality of datasets available. As a result, the current lack of data and the high cost of collecting it in real-world scenarios further limit the applicability of LMs. In this paper, we leverage the powerful generation capabilities of LLMs to enhance pre-trained LMs. Specifically, we use LLMs to generate domain-specific data, thereby improving the performance of pre-trained LMs on the target tasks. We conduct experiments by combining different LLMs in our generation phase and introducing various LMs to learn from the LLM-generated data. Then, we compare the performance of these LMs before and after learning the data. We find that LLM-generated data significantly enhances the performance of LMs. The improvement can reach up to 58.36% for fault localization and up to 6.09% for clone detection. Our study highlights that using LLMs to generate data for LMs can improve performance by a large margin.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Multitask-Based Evaluation of Open-Source LLM on Software VulnerabilityIEEE Transactions on Software Engineering, 2024
- ThinkRepair: Self-Directed Automated Program Repair2024
- Prompting Is All You Need: Automated Android Bug Replay with Large Language Models2024
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X2023
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity2023
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation2021
- Unified Pre-training for Program Understanding and Generation2021
- Detecting Code Clones with Graph Neural Network and Flow-Augmented Abstract Syntax Tree2020
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages2020
- Deep Learning-Based Fault Localization with Contextual InformationIEICE Transactions on Information and Systems, 2017