Planning to Guide LLM for Code Coverage Prediction
Open Access
- 14 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 24-34
Abstract
Code coverage serves as a crucial metric to assess testing effectiveness, measuring the degree to which a test suite exercises different facets of the code, such as statements, branches, or paths. Despite its significance, coverage profilers necessitate access to the entire codebase, constraining their usefulness in situations where the code is incomplete or execution is not feasible, and even cost-prohibitive. In this paper, we present CodePilot, a plan-based prompting approach grounded in program semantics, which collaborates with a Large Language Model (LLM) to enhance code coverage prediction. To address the intricacies of predicting code coverage, CodePilot employs planning by discerning various types of statements in an execution flow. Planning empowers GPT to autonomously generate plans based on guided examples, and then CodePilot prompts the GPT model to predict code coverage (Action) based on the plan it generated (Reasoning). Our experiments evaluating CodePilot demonstrate high accuracy, achieving up to 55% in exact-match and 89% in statement-match. It performs relatively better than the baselines, achieving up to 33% and 19% relatively higher in those metrics. We also showed that due to highly accurate plans (90%), GPT model predicts better code coverage. Moreover, we show CodePilot's utility in correctly predicting the least covered statements.Keywords
Funding Information
- National Science Foundation (CNS-2120386)
- National Security Agency (NCAE-C-002-2021)
This publication has 21 references indexed in Scilit:
- Reachable Coverage: Estimating Saturation in Fuzzing2023
- Fuzzing: Challenges and ReflectionsIEEE Software, 2020
- Full-Speed Fuzzing: Reducing Fuzzing Overhead through Coverage-Guided Tracing2019
- An automated approach to estimating code coverage measures via execution logs2018
- Code coverage for suite evaluation by developers2014
- Is Branch Coverage a Good Measure of Testing Effectiveness?2012
- Leveraging disposable instrumentation to reduce coverage collection overheadSoftware Testing, Verification and Reliability, 2006
- Demand-driven structural testing with dynamic instrumentation2005
- Efficient online computation of statement coverageJournal of Systems and Software, 2005
- Residual test coverage monitoring1999