Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering
Preprint
Abstract
Code snippet adaptation is a fundamental activity in the software development process. Unlike code generation, code snippet adaptation is not a "free creation", which requires developers to tailor a given code snippet in order to fit specific requirements and the code context. Recently, large language models (LLMs) have confirmed their effectiveness in the code generation task with promising results. However, their performance on adaptation, a reuse-oriented and context-dependent code change prediction task, is still unclear. To bridge this gap, we conduct an empirical study to investigate the performance and issues of LLMs on the adaptation task. We first evaluate the adaptation performances of three popular LLMs and compare them to the code generation task. Our result indicates that their adaptation ability is weaker than generation, with a nearly 15% decrease on pass@1 and more context-related errors. By manually inspecting 200 cases, we further investigate the causes of LLMs' sub-optimal performance, which can be classified into three categories, i.e., Unclear Requirement, Requirement Misalignment and Context Misapplication. Based on the above empirical research, we propose an interactive prompting approach to eliciting LLMs' adaptation ability. Experimental result reveals that our approach greatly improve LLMs' adaptation performance. The best-performing Human-LLM interaction successfully solves 159 out of the 202 identified defects and improves the pass@1 and pass@5 by over 40% compared to the initial instruction-based prompt. Considering human efforts, we suggest multi-agent interaction as a trade-off, which can achieve comparable performance with excellent generalization ability. We deem that our approach could provide methodological assistance for autonomous code snippet reuse and adaptation with LLMs.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- A study on prompt design, advantages and limitations of ChatGPT for deep learning program repairAutomated Software Engineering, 2025
- ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design2024
- Fusing Code SearchersIEEE Transactions on Software Engineering, 2024
- How Do Developers Adapt Code Snippets to Their Contexts? An Empirical Study of Context-Based Code Snippet AdaptationsIEEE Transactions on Software Engineering, 2024
- What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?2023
- Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability2023
- An Analysis of the Automatic Bug Fixing Performance of ChatGPT2023
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm2021
- Usage and attribution of Stack Overflow code snippets in GitHub projectsEmpirical Software Engineering, 2018
- Writing Code to Prototype, Ideate, and DiscoverIEEE Software, 2009