exLong: Generating Exceptional Behavior Tests with Large Language Models
Authors:
- 26 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1462-1474
Abstract
Many popular programming languages, including C#, Java, and Python, support exceptions. Exceptions are thrown during program execution if an unwanted event happens, e.g., a method is invoked with an illegal argument value. Software developers write exceptional behavior tests (EBTs) to check that their code detects unwanted events and throws appropriate exceptions. Prior research studies have shown the importance of EBTs, but those studies also highlighted that developers put most of their efforts on “happy paths”, e.g., paths without unwanted events. To help developers fill the gap, we present the first framework, dubbed ExLoNG, that automatically generates EBTs. ExLONG is a large language model instruction fine-tuned from CodeLlama and embeds reasoning about traces that lead to throw statements, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. We compare ExLONG with the state-of-the-art models for test generation (CAT-LM) and one of the strongest foundation models (GPT-4o), as well as with analysis-based tools for test generation (Randoop and EvoSuite). Our results show that ExLONG outperforms existing models and tools. Furthermore, we contributed several pull requests to open-source projects and 23 EBTs generated by ExLONG were already accepted.Keywords
All Related Versions
Funding Information
- AST (2421782)
- Simons Foundation (MPS-AI-00010515)
- NSF
This publication has 46 references indexed in Scilit:
- Optimal test case generation for boundary value analysisSoftware Quality Journal, 2024
- CAT-LM Training Language Models on Aligned Code And Tests2023
- When to Say What: Learning to Find Condition-Message Inconsistencies2023
- Test smells 20 years later: detectability, validity, and reliabilityEmpirical Software Engineering, 2022
- Assessing exception handling testing practices in open-source librariesEmpirical Software Engineering, 2021
- Automatically Generating Precise Oracles from Structured Natural Language Specifications2019
- Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs2011
- The Daikon system for dynamic detection of likely invariantsScience of Computer Programming, 2007
- Feedback-Directed Random Test Generation29th International Conference on Software Engineering (ICSE'07), 2007
- Automated test-data generation for exception conditionsSoftware: Practice and Experience, 2000