From Executable to Demonstrable: Synthesizing API Usage Showcases for Research Repositories
Open Access
Abstract
A script that runs a research repository is not the same as a script that shows how to use it. Test generation targets structural coverage, and work on executability targets whether a repository can be made to run. Neither yields what a new user of a research library needs, which is a worked example. We define demonstrativeness by three explicit predicates: in-process contact with a public non-driver symbol, use of a constructed argument, and output that depends on the returned value. We record these static predicates separately from environment construction and trial execution, preventing dependency-blocked repositories from being labelled runtime successes or failures. Reaching a library’s interface requires a target model that entry-point selection cannot express; we derive symbol-level targets by Horn clauses over signatures, exports, docstrings, and documentation references. Generated scripts are read back after every successful execution, and a vacuous script can be repaired without discarding the best executable fallback. In a paired, single-run evaluation on 90 repositories, of which 24 had buildable environments, the full configuration produced 11 passing showcases against 2 for an entry-point-only configuration and made static API contact in 15 buildable environments against 7, while both retained Tier-1 success on the same 21; exact McNemar tests on the per-repository records give and . Corpus-wide, the entry-point-only configuration attempted generation on 24 repositories and the full configuration on 89, with API contact in 7 and 70; that contrast is largely by design, since only the full configuration has a generation path in a dependency-blocked environment. Because the full configuration bundles API targets, read-back, and vacuity repair, these figures establish a system-level contrast, not the isolated causal effect of any component.
Keywords
Funding Information
- Constructor Knowledge Labs
This publication has 15 references indexed in Scilit:
- An Empirical Evaluation of Using Large Language Models for Automated Unit Test GenerationIEEE Transactions on Software Engineering, 2023
- CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models2023
- Pynguin: Automated Unit Test Generation for Python2022
- A large-scale study on research code quality and executionScientific Data, 2022
- An empirical analysis of journal policy effectiveness for computational reproducibilityProceedings of the National Academy of Sciences of the United States of America, 2018
- Repeatability in computer systems researchCommunications of the ACM, 2016
- An introduction to Docker for reproducible researchACM SIGOPS Operating Systems Review, 2015
- Synthesizing API usage examples2012
- EvoSuite2011
- Feedback-Directed Random Test Generation29th International Conference on Software Engineering (ICSE'07), 2007