CDBench: Benchmarking the mutation testing capabilities of LLMs with code defenders
Open Access
- 13 June 2026
- journal article
- research article
- Published by Springer Nature in Empirical Software Engineering
- Vol. 31 (6) , 166
Abstract
Most traditional benchmarks for evaluating Large Language Models (LLMs) in software development suffer from a narrow focus, high risks of data contamination, and static difficulty levels that fail to keep pace with rapid model evolution. To address these limitations, we introduce CDBench, a novel zero-sum benchmark based on the Code Defenders mutation testing game. By pitting models against each other in a competitive environment—where “attackers” introduce code mutations and “defenders” create tests to detect them—CDBench establishes a dynamic difficulty curve that scales naturally without human intervention. Our experiments reveal that while LLMs can generate diverse mutations, they often struggle with code validity; nevertheless, the framework effectively distinguishes model capabilities, highlighting the superior test generation of models like Gemini 2.5 Pro while exposing the instruction-following limitations of reasoning models. These findings demonstrate that zero-sum games offer a viable, contamination-resistant solution to the stagnation of current evaluation methodologies.Keywords
Funding Information
- Universität Passau
This publication has 65 references indexed in Scilit:
- A new perspective on the competent programmer hypothesis through the reproduction of real faults with repeated mutationsSoftware Testing, Verification and Reliability, 2024
- Generating Python Mutants From Bug Fixes Using Neural Machine TranslationIEEE Access, 2023
- An empirical study of automated unit test generation for PythonEmpirical Software Engineering, 2023
- JUGE: An infrastructure for benchmarking Java unit test generatorsSoftware Testing, Verification and Reliability, 2022
- Sampling in software engineering research: a critical review and guidelinesEmpirical Software Engineering, 2022
- Mutation Testing Advances: An Analysis and Survey2019
- Experimentation in Software Engineering2012
- Guidelines for conducting and reporting case study research in software engineeringEmpirical Software Engineering, 2008
- JCrasher: an automatic robustness tester for JavaSoftware: Practice and Experience, 2004
- Hints on Test Data Selection: Help for the Practicing ProgrammerComputer, 1978