Towards Quantum Program Bug Datasets and Benchmarking
Open Access
- 8 August 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 9338-9348
Abstract
The growing maturity of quantum computing has led to increasingly complex software ecosystems spanning frameworks, compilers, simulators, and application-level programs. Despite this rapid growth, the community still lacks a realistic benchmark for evaluating automated quantum bug fixing under quantum-specific correctness requirements, such as stochastic outputs, tolerance-based equivalence, and environment-sensitive execution. To bridge this gap, we present QBench, an end-to-end benchmark for evaluating automated repair of real-world quantum software bugs. QBench consists of two complementary tracks: RepoTrack, which targets defects in quantum software infrastructure and repositories, and ScriptTrack, which targets bugs in user-facing quantum programs and experimental workflows. Each benchmark instance provides a reproducible bug-fixing task together with the execution environment, correctness oracle, and evaluation protocol required to validate candidate repairs, enabling standardized assessment of repair systems under realistic development settings. QBench supports deterministic, tolerance-aware, and probabilistic correctness oracles, enabling reliable evaluation under realistic quantum execution semantics. Using QBench, we conduct a large-scale empirical study of automated quantum software repair and analyze the challenges faced by representative repair approaches. Our results show that stochastic correctness criteria, environment fragility, and cross-layer defects remain major obstacles to successful repair, highlighting the unique difficulties of maintaining and repairing quantum software.Keywords
Funding Information
- Ningbo Yongjiang Talent Programme (-)
- CPS-Yangtze Delta Region Industrial Innovation Center of Quantum and Information Technology-MindSpore Quantum Open Fund (-)
This publication has 37 references indexed in Scilit:
- AutoFid: Adaptive and Noise-Aware Fidelity Measurement for Quantum Programs via Circuit Graph Analysis2025
- Fortuna: Towards Efficient Selection of High-Fidelity Link for Quantum Network in the Wild2025
- Moirai: Optimizing Quantum Serverless Function Orchestration via Device Allocation and Circuit Deployment2024
- An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair2023
- Automated Program Repair in the Era of Large Pre-trained Language Models2023
- MorphQ: Metamorphic Testing of the Qiskit Quantum Computing Platform2023
- QDiff: Differential Testing of Quantum Software Stacks2021
- Bugs4Q: A Benchmark of Real Bugs for Quantum Programs2021
- Automatic patch generation learned from human-written patches2013
- Automatically finding patches using genetic programming2009