STRUT: Structured Seed Case Guided Unit Test Generation for C Programs using LLMs
Open Access
- 22 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2 (ISSTA) , 2113-2135
Abstract
Unit testing plays a crucial role in bug detection and ensuring software correctness. It helps developers identify errors early in development, thereby reducing software defects. In recent years, large language models (LLMs) have demonstrated significant potential in automating unit test generation. However, using LLMs to generate unit tests faces many challenges. 1) The execution pass rate of the test cases generated by LLMs is low. 2) The test case coverage is inadequate, making it challenging to detect potential risks in the code. 3) Current research methods primarily focus on languages such as Java and Python, while studies on C programming are scarce, despite its importance in the real world. To address these challenges, we propose STRUT, a novel unit test generation method. STRUT utilizes structured test cases as a bridge between complex programming languages and LLMs. Instead of directly generating test code, STRUT guides LLMs to produce structured test cases, thereby alleviating the limitations of LLMs when generating code for programming languages with complex features. First, STRUT analyzes the context of focal methods and constructs structured seed test cases for them. These seed test cases then guide LLMs to generate a set of structured test cases. Subsequently, a rule-based approach is employed to convert the structured set of test cases into executable test code. We conducted a comprehensive evaluation of STRUT, which achieved an impressive execution pass rate of 96.01%, along with 77.67% line coverage and 63.60% branch coverage. This performance significantly surpasses that of the LLMs-based baseline methods and the symbolic execution tool SunwiseAUnit. These results highlight STRUT's superior capability in generating high-quality unit test cases by leveraging the strengths of LLMs while addressing their inherent limitations.Keywords
Funding Information
- National Key Research and Development Program of China (2022YFB4501900)
- Natural Science Basic Research Program of Shaanxi (2024JC-YBQN-0732)
This publication has 35 references indexed in Scilit:
- A3Test: Assertion-Augmented Automated Test case generationInformation and Software Technology, 2024
- An Empirical Evaluation of Using Large Language Models for Automated Unit Test GenerationIEEE Transactions on Software Engineering, 2023
- CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models2023
- CITRUS: Automated Unit Testing Tool for Real-world C++ Programs2022
- GRT: Program-Analysis-Guided Random Testing (T)2015
- Achievements, Open Problems and Challenges for Search Based Software Testing2015
- Understanding the Test Automation Culture of App Developers2015
- A Survey on Unit Testing Practices and Problems2014
- Instance Generator and Problem Representation to Improve Object Oriented Code CoverageIEEE Transactions on Software Engineering, 2014
- A survey of unit testing practicesIEEE Software, 2006