Andreas Zeller

Understanding the semantics of program code is a fundamental problem in software engineering, posing continuous challenges for testing and analysis, as well as for AI agents. We introduce neural program modeling : an approach that learns the relationship between input features and execution features (such as traces or coverage) from pairs of generated system inputs and monitored execution data. The resulting program-specific model can predict system inputs that trigger desired program behaviors—in seconds, without executing the program. It can also predict program traces and coverage for a given input, again without running the program. Once trained, such a model acts as a proxy for the program under test, supporting a range of tasks: (1) Predicting inputs for specific execution features, guiding test generators and developers; (2) Replacing the original program during search-based testing or fuzzing, reducing costly or risky executions; (3) Predicting which program parts a test input will execute, guiding test selection ; (4) Predicting which input will trigger a specific behavior, serving as an input filter to prevent undesired outcomes. To our knowledge, this is the first work that trains reversible, program-specific neural machine translation models to both predict execution behavior from inputs and synthesize inputs to achieve the requested behavior in real-world programs. In our evaluation of real-world programs, including Bottle and the Google re2 library, we show that neural program modeling can predict execution features for unseen inputs, achieving similarity ratios ranging from 91.57% to 98.59%. We can also predict program inputs that trigger specified behaviors with accuracy and recall up to 100% after validation and refinement. All predictions can be automatically validated and refined, either by retraining the model on new input-trace pairs or by using a backpropagation loop to adjust its parameters. Model training and inference are resource-efficient and can be performed on commodity hardware.

Statistical fault localization (SFL) techniques use test coverage to rank suspicious program elements. However, they often suffer from low precision when tests contain multiple assertions or unrelated code that obscures the relationship between test outcomes and faulty code. Test case purification addresses this problem by transforming complex test cases into simplified, single-assertion variants. We present PYURIFY, a tool that implements test case purification for Python through two steps: first, atomization splits each test with multiple assertions into separate single-assertion tests; second, dynamic program slicing removes code irrelevant to each assertion. This purification process produces cleaner test spectra, enabling SFL techniques to pinpoint faulty code more accurately. PYURIFY provides both a command-line interface and a Python API for seamless integration with existing testing and fault localization workflows. Our evaluation of real Python bugs demonstrates that test case purification consistently improves fault localization accuracy.

Even in the age of AI, generating comprehensive test inputs for software systems remains a challenge, particularly for interactive and reactive systems, which require entire interaction sequences to reach the desired states or to cover missing behavior. While modern fuzzers are effective at generating random inputs that test the robustness of input processors, they still struggle to produce or mutate complex inputs and interactions. LLM-based systems, on the other hand, can generate small example inputs, but fail to systematically explore the space of inputs and interactions as would be required for comprehensive testing. In this tutorial, we introduce test generation with fandango, a modern specification-based generator of test inputs and interactions. Based on a spec file that defines the structure and properties of the program input, fandango produces inputs that are syntactically and semantically correct, and systematically cover the input space of the program under test. In three interactive sessions, we (1) introduce language-based testing, (2) demonstrate how to use constraints to define desired properties, and then (3) delve into full-fledged protocol testing, producing entire interaction sequences between fandango and the network components under test. Participants are expected to bring a basic understanding of software testing; knowledge of python is a plus, but not required. At the end of the tutorial, they will (1) be able to specify and test formats for inputs and interactions, (2) produce comprehensive test suites using language-based techniques, and (3) understand the trade-offs and practical applications of language-based testing.

In the past years, large language models (LLMs) have demonstrated remarkable progress in code generation. However, their ability to reason about program behavior remains an open challenge—an ability that is relevant for applications including reverse engineering, debugging, secure code generation, test-driven synthesis, input reconstruction, reverse fuzzing, behavioral monitoring, and safe execution modeling. To study this ability, we examine the capacity of LLMs to reason about the semantics of code—specifically, their ability to relate code, its inputs, and its outputs to each other. To this end, we investigate whether and how well LLMs can predict one of these three components given the other two—that is, (1) predict the input given code and output, (2) predict the output given code and input, and (3) predict the code given input and output. This way, we assess how well LLMs can reason about and understand the underlying relationships that govern program execution. We construct four datasets covering string processing, array operations, and coding challenges in JavaScript and Python to evaluate diverse program-understanding capabilities, incorporating various code mutation techniques to increase complexity. In our evaluation on tasks covering string processing, array operations, and coding challenges, we find that closed-weight models achieve the strongest performance across all datasets, including perfect input recovery on deterministic string tasks. Across tasks, output prediction is comparatively stable, whereas code prediction remains the hardest setting and often fails for smaller models. Finally, cross-codebase transfer is feasible, especially for input prediction, but highly sensitive to model capacity and fine-tuning strategy.

When a program fails, statistical fault localization (SFL) provides important debugging hints by identifying the locations whose execution most correlates with failures. However, such correlations can be weakened if a test contains both _passing_ and _failing_ assertions, creating ambiguous and misleading associations. Likewise, if multiple lines correlate with the same strength, SFL provides little guidance to disambiguate between them. This paper proposes a novel proximity-based weighting scheme for SFL that assigns different _weights_ to locations in the test subject based on temporal proximity to a failure. The more recently a subject line is executed before the test fails, the higher its weight. We operationalize a known heuristic into a lightweight statistical form compatible with existing SFL formulas. Our approach applies to _any test_, from simple single-line tests (where it preserves SFL behavior), to single-assertion tests with multiple lines (where it benefits from temporal proximity), to complex multi-assertion tests (where it provides the most benefit by distinguishing failing from passing assertions). Once computed, the weights can be integrated into any existing SFL technique. Our evaluation of proximity-weighted fault localization on 310~real-world programs shows that it consistently outperforms fault localization techniques across all test types. Proximity-weighted fault localization shows per-subject relative improvements of 200%–400%, meaning that, for a typical subject, it provides 3 to 5 times the baseline effectiveness. These improvements represent substantial gains over baseline techniques. Our approach can be integrated into existing fault localization techniques to improve performance, making it a valuable addition to automated debugging.

We introduce Modelizer —a novel framework that, given a black-box program, learns a model from its input/output behavior using neural machine translation algorithms. The resulting model mocks the original program: Given an input, the model predicts the output that would have been produced by the program. However, the model is also reversible —that is, the model can predict the input that would have produced a given output. Finally, the model is differentiable and can be efficiently restricted to predict only a certain aspect of the program behavior. Modelizer uses grammars to synthesize and inputs and unsupervised tokenizers to decompose the resulting outputs, allowing it to learn sequence-to-sequence associations between token streams. Other than input grammars, Modelizer only requires the ability to execute the program. The resulting models are small, requiring fewer than 6.3 million parameters for languages such as Markdown or HTML; and they are accurate, achieving up to 95.4% accuracy and a BLEU score of 0.98 with standard error of 0.04 in mocking real-world applications. As it learns from and predicts executions rather than code, Modelizer departs from the LLM-centric research trend, opening new opportunities for program-specific models that are fully tuned toward individual programs. Indeed, we foresee several applications of these models, especially as the output of the program can be any aspect of program behavior. Beyond mocking and predicting program behavior, the models can also synthesize inputs that are likely to produce a particular behavior, such as failures or coverage, thus assisting in program understanding and maintenance.