Cerberus: Multi-Agent Reasoning and Coverage-Guided Exploration for Static Detection of Runtime Errors
Preprint
Abstract
In several software development scenarios, it is desirable to detect runtime errors and exceptions in code snippets without actual execution. A typical example is to detect runtime exceptions in online code snippets before integrating them into a codebase. In this paper, we propose Cerberus, a novel predictive, execution-free coverage-guided testing framework. Cerberus uses LLMs to generate the inputs that trigger runtime errors and to perform code coverage prediction and error detection without code execution. With a two-phase feedback loop, Cerberus first aims to both increasing code coverage and detecting runtime errors, then shifts to focus only detecting runtime errors when the coverage reaches 100% or its maximum, enabling it to perform better than prompting the LLMs for both purposes. Our empirical evaluation demonstrates that Cerberus performs better than conventional and learning-based testing frameworks for (in)complete code snippets by generating high-coverage test cases more efficiently, leading to the discovery of more runtime errors.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models2023
- Improving bug detection via context-based code representation learning and attention-based neural networksProceedings of the ACM on Programming Languages, 2019
- JQF: coverage-guided property-based testing in Java2019
- Analyzing and Supporting Adaptation of Online Code Examples2019
- POSTER2017
- Directed Greybox Fuzzing2017
- Improving your software using static analysis to find bugs2006
- CUTE2005
- DART2005
- The value of slicing while debuggingScience of Computer Programming, 2001