LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework
Authors:
- 16 November 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2401-2413
Abstract
Unit testing is an essential but resource-intensive step in software development, ensuring individual code units function correctly. This paper introduces AgoneTest, an automated evaluation framework for Large Language Model-generated (LLM) unit tests in Java. AgoneTest does not aim to propose a novel test generation algorithm; rather, it supports researchers and developers in comparing different LLMs and prompting strategies through a standardized end-to-end evaluation pipeline under realistic conditions. We introduce the CLASSES2TEST dataset, which maps Java classes under test to their corresponding test classes, and a framework that integrates advanced evaluation metrics, such as mutation score and test smells, for a comprehensive assessment. Experimental results show that, for the subset of tests that compile, LLM-generated tests can match or exceed human-written tests in terms of coverage and defect detection. Our findings also demonstrate that enhanced prompting strategies contribute to test quality. AgoneTest clarifies the potential of LLMs in software testing and offers insights for future improvements in model design, prompt engineering, and testing practices.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- A3Test: Assertion-Augmented Automated Test case generationInformation and Software Technology, 2024
- Effective test generation using pre-trained Large Language Models and mutation testingInformation and Software Technology, 2024
- JUGE: An infrastructure for benchmarking Java unit test generatorsSoftware Testing, Verification and Reliability, 2022
- Basic block coverage for search-based unit testing and crash reproductionEmpirical Software Engineering, 2022
- Scented since the beginning: On the diffuseness of test smells in automatically generated test codeJournal of Systems and Software, 2019
- An Experimental Study on Flakiness and Fragility of Randoop Regression Test Suites2019
- Mutation Testing Advances: An Analysis and Survey2019
- GRT: Program-Analysis-Guided Random Testing (T)2015
- JCrasher: an automatic robustness tester for JavaSoftware: Practice and Experience, 2004
- The Test Automation Manifesto2003